Google Cloud BigQuery Import does not work in application project

Question

I used the following code to create an application engine project to move data from a Google cloud to a bigquery table.

import argparse import time import uuid from google.cloud import bigquery def load_data_from_gcs(dataset_name, table_name, source): bigquery_client = bigquery.Client() dataset = bigquery_client.dataset(dataset_name) table = dataset.table(table_name) job_name = str(uuid.uuid4()) job = bigquery_client.load_table_from_storage( job_name, table, source) job.begin() wait_for_job(job) print('Loaded {} rows into {}:{}.'.format( job.output_rows, dataset_name, table_name)) def wait_for_job(job): while True: job.reload() if job.state == 'DONE': if job.error_result: raise RuntimeError(job.error_result) return time.sleep(1) if __name__ == '__main__': parser = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) parser.add_argument('dataset_name') parser.add_argument('table_name') parser.add_argument( 'source', help='The Google Cloud Storage object to load. Must be in ' 'the format gs://bucket_name/object_name') args = parser.parse_args() load_data_from_gcs( args.dataset_name, args.table_name, args.source) 

I also changed the default app.yaml file as the above file and deleted the webapp2 library entry, and my app.yaml file looks like this:

 application: gcstobq version: 1 runtime: python27 api_version: 1 threadsafe: yes handlers: - url: /favicon\.ico static_files: favicon.ico upload: favicon\.ico - url: .* script: main.app 

Since I am new to python and the application engine, I don’t know whether to include libraries in the main.py file in app.yaml, and if I need to run this application using the command line tool.

Please let me know if I missed something here?

Answer 1

Google Cloud uses the new Python namespace format (if you look at the source, you will notice that there is no __init__.py in the directory structure). This has been changed in Python 3.3 with PEP-420

Fortunately, in Python 2.7 you can easily fix this by avoiding implicit imports. Just add this to the very top of your file (before any other import) to get Python 3 behavior:

from __future__ import absolute_import