How to upload data from the GAE online data warehouse to a local development server?

Question

I previously used the approach described in the GAE docs to load backups of my objects into a real data warehouse.

I currently have an entity view csv file that I got by writing bulkloader.yaml and using this command:

 appcfg.py download_data --config_file=bulkloader.yaml --filename=users.csv --kind=Permission --url=http://your_app_id.appspot.com/_ah/remote_api 

I also have a sql3 dump file that I got with the command:

 appcfg.py download_data --kind=<kind> --url=http://your_app_id.appspot.com/_ah/remote_api --filename=<data-filename> 

Now, if I try this command:

 appcfg.py upload_data --url=http://your_app_id.appspot.com/_ah/remote_api --kind=<kind> --filename=<data-filename> 

Replacing the localhost: 8080 URL, it asks me for the username / password. Now even if you provide a false username ( [e-mail] ) in http://localhost:8080/_ah/remote_api and check the "admin" box, it always gives me an authentication error.

Another alternative mentioned in the docs uses the following:

 appcfg.py upload_data --config_file=album_loader.py --filename=album_data.csv --kind=Album --url=http://localhost:8080/_ah/remote_api <app-directory> 

I wrote a bootloader and tried it, it also asks for a username and password, but it accepts something here. The output is as follows:

 /usr/local/google_appengine/google/appengine/api/search/search.py:232: UserWarning: DocumentOperationResult._code is deprecated. Use OperationResult._code instead. 'Use OperationResult.%s instead.' % (name, name)) /usr/local/google_appengine/google/appengine/api/search/search.py:232: UserWarning: DocumentOperationResult._CODES is deprecated. Use OperationResult._CODES instead. 'Use OperationResult.%s instead.' % (name, name)) Application: knowledgetestgame Uploading data records. [INFO ] Logging to bulkloader-log-20121113.210613 [INFO ] Throttling transfers: [INFO ] Bandwidth: 250000 bytes/second [INFO ] HTTP connections: 8/second [INFO ] Entities inserted/fetched/modified: 20/second [INFO ] Batch Size: 10 [INFO ] Opening database: bulkloader-progress-20121113.210613.sql3 Please enter login credentials for localhost Email: [e-mail] Password for [e-mail] : [INFO ] Connecting to localhost:8080/_ah/remote_api [INFO ] Starting import; maximum 10 entities per post [ERROR ] [WorkerThread-4] WorkerThread: Traceback (most recent call last): File "/usr/local/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 176, in WorkOnItems status, instruction = item.PerformWork(self.__thread_pool) File "/usr/local/google_appengine/google/appengine/tools/bulkloader.py", line 764, in PerformWork transfer_time = self._TransferItem(thread_pool) File "/usr/local/google_appengine/google/appengine/tools/bulkloader.py", line 933, in _TransferItem self.content = self.request_manager.EncodeContent(self.rows) File "/usr/local/google_appengine/google/appengine/tools/bulkloader.py", line 1394, in EncodeContent entity = loader.create_entity(values, key_name=key, parent=parent) File "/usr/local/google_appengine/google/appengine/tools/bulkloader.py", line 2728, in create_entity (len(self.__properties), len(values))) AssertionError: Expected 17 columns, found 18. [INFO ] [WorkerThread-5] Backing off due to errors: 1.0 seconds [INFO ] Unexpected thread death: WorkerThread-4 [INFO ] An error occurred. Shutting down... [ERROR ] Error in WorkerThread-4: Expected 17 columns, found 18. [INFO ] 980 entities total, 0 previously transferred [INFO ] 0 entities (278 bytes) transferred in 5.9 seconds [INFO ] Some entities not successfully transferred 

I have ~ 4000 objects in total, it says that 980 are transferred, but in fact I check the local data store and I can not find any of them.

The loader is used below (I used NDB for the Guess object)

 import datetime from google.appengine.ext import db from google.appengine.tools import bulkloader from google.appengine.ext.ndb import key class Guess(db.Model): pass class GuessLoader(bulkloader.Loader): def __init__(self): bulkloader.Loader.__init__(self, 'Guess', [('selectedAssociation', lambda x: x.decode('utf-8')), ('suggestionsList', lambda x: x.decode('utf-8')), ('associationIndexInList', int), ('timeEntered', lambda x: datetime.datetime.strptime(x, '%m/%d/%Y').date()), ('rank', int), ('topicName', lambda x: x.decode('utf-8')), ('topic', int), ('player', int), ('game', int), ('guessString', lambda x: x.decode('utf-8')), ('guessTime', lambda x: datetime.datetime.strptime(x, '%m/%d/%Y').date()), ('accountType', lambda x: x.decode('utf-8')), ('nthGuess', int), ('score', float), ('cutByRoundEnd', bool), ('suggestionsListDelay', int), ('occurrences', float) ]) loaders = [GuessLoader] 

Edit: I just noticed this part in the error message [ERROR ] Error in WorkerThread-0: Expected 17 columns, found 18. , while in fact I just looked at the whole csv file and made sure that each line has 18 columns. I checked the bootloader and found that I was missing the key column, I gave it an int type, but that does not work.

Answer 1

If you are having authentication problems, enter the following in appengine_config.py:

 if os.environ.get('SERVER_SOFTWARE','').startswith('Development'): remoteapi_CUSTOM_ENVIRONMENT_AUTHENTICATION = ( 'REMOTE_ADDR', ['127.0.0.1']) 

then run

 appcfg.py download_data --url=http://APPNAME.appspot.com/_ah/remote_api --filename=dump --kind=EntityName appcfg.py upload_data --url=http://localhost:8080/_ah/remote_api --filename=dump --application=dev~APPNAME 

Answer 2

Try simply pressing Enter (no username / password). This seemed to be a trick for me. My command (wrapped in a bash script to prevent import errors that I sometimes get):

 #!/bin/bash # Modify path export PYTHONPATH=$PYTHONPATH:. # Load data python /path/to/app/config/appcfg.py upload_data \ --config_file=<my_loader.py> \ --filename=<output.csv> \ --kind=<kind> \ --application=dev~<application_id> \ --url=http://localhost:8088/_ah/remote_api ./ 

When prompted for Email , I press Enter and everything is uploaded to the dev server. I do not use NDB in this case, although I do not believe that this should matter.