How to import BigQuery in AppEngine for Python

Question

I am trying to run a BigQuery query from Google AppEngine (deployed) using Python 2.7, but I see this error in StackDriver Error Reporting:

ImportError: no module named cloud

This is my code (main.py):

from __future__ import absolute_import import webapp2 from google.cloud import bigquery class MainPage(webapp2.RequestHandler): def get(self): # Instantiates a client bigquery_client = bigquery.Client() # The name for the new dataset dataset_name = 'my_new_set' # Prepares the new dataset dataset = bigquery_client.dataset(dataset_name) # Creates the new dataset dataset.create() # Remove unwanted chars #self.response.write(str(container)) app = webapp2.WSGIApplication([ ('/', MainPage), ], debug=True) 

This is my (app.yaml):

 runtime: python27 api_version: 1 threadsafe: true handlers: - url: /.* script: main.app 

The error message makes me assume that the BigQuery library is not being imported. However, if this code is deployed to AppEngine, should the library not already be installed in AppEngine by default?

---

Trying to solve a problem

Attempt # 1

I found this post that relates to a similar issue. It was suggested to add this line to the top of the file. I added a line to my file, but the problem still exists:

 from __future__ import absolute_import 

Source: No module named cloud when using a lot of google.cloud requests

Attempt # 2

I installed the BigQuery client locally on my laptop:

 pip install google-cloud-bigquery==0.22.1 

I also installed the same client in the "lib" folder to load it into AppEngine after it was deployed:

 pip install --target='lib' google-cloud-bigquery==0.22.1 

In the latter case, you also need to create a file called "appengine_config.py" with this content:

 # appengine_config.py from google.appengine.ext import vendor # Add any libraries install in the "lib" folder. vendor.add('lib') 

Source: https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27

However, this attempt did not work either. The error message has changed to the following:

 *File "/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/httplib2/__init__.py", line 352: print('%s:' % h, end=' ', file=self._fp) ^ SyntaxError: invalid syntax at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google_auth_httplib2.py:23) at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/_helpers.py:31) at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/bigquery/_helpers.py:21) at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/bigquery/__init__.py:26) at get (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/main.py:75) at dispatch (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:545) at dispatch (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:547) at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1077) at default_dispatcher (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1253) at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1505) at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1511)* 

How can I import the BigQuery library correctly in AppEngine (deployed)?

Thank you for your help.

Answer 1

The following solution worked for me without using from __future__ import absolute_import . There are three main steps.

1. Copy google-api-python-client and google-cloud to the project folder

Despite the fact that this sounds inconsistent, according to the documentation

[...] Python client libraries are not installed in the Python App Engine runtime, so they should be presented in your application in the same way as third-party libraries.

So, to use google.cloud , you need to copy the library code into the project source directory. The library code along with the application code is uploaded to App Engine.

To copy library code to a project:

• Create a directory (e.g. lib /) to store third-party libraries in the project root folder

   mkdir lib 

• Copy the google-api-python-client and google-cloud libraries into the folder you just created. I am using pip in the following example.

   pip install -t lib/ --upgrade google-api-python-client pip install -t lib/ --upgrade google-cloud 

2. Associate installed libraries with the application

• Create a file called appengine_config.py in the same folder as your app.yaml file

• Modify appengine_config.py and include the following code

   # appengine_config.py from google.appengine.ext import vendor # Add any libraries install in the "lib" folder. vendor.add('lib') 

3. Include the added libraries in requirements.txt

• Edit the requirements.txt file and include the names of the added libraries

   # other requirements google-api-python-client google-cloud 

Now you can use from google.cloud import bigquery without problems after deploying your application.

See using third-party libraries for more information.