Import large amounts of data into the Google App Engine datastore at a time

Question

I have a large CSV file about 10 MB in size, which contains all the data that needs to be imported into the Google App Engine DataStore. I tried the following import execution approaches, but all the time it failed halfway.

• Import using URL match command and then execute URL failed due to request timeout ...
• Import using cron job but got DeadlineExceededError ...
• Import using remort_api_shell, but the operation timed out.

Can you suggest me and approve (using fictitious data that you can imagine) how to do this ... An offer with a code will be more useful ..

** I am using Python and the Google web application platform to develop this application.

Answer 1

you can send line by line. using the built-in bootloader.

http://code.google.com/appengine/docs/python/tools/uploadingdata.html

This is a good article.

and here is my contactloader.py, which I used 2 years ago for reference. it is more sophisticated since I used the latter, but still .....

import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader

class Contact(db.Model):

    date = db.DateTimeProperty(auto_now_add=True)

    owner = db.StringProperty()

    companyname = db.StringProperty()

    companyemail = db.EmailProperty()

def myfunc(x):
    temp = x.split(":mailto:")
    if len(temp) > 0:
        temp = temp[-1].split(":")
    else:
        return "defaultvalue"
    if len(temp) > 0:
        temp = temp[0]
    else:
        return "defaultvalue"
    temp = temp.split("<1>")[0]
    if temp is None or len(temp) < 5:
        return "defaultvalue"
    return temp

def mysecfunc(x):
    return x.split("<0>")[0]

class ContactLoader(bulkloader.Loader):
    def __init__(self):
        bulkloader.Loader.__init__(self, 'Contact',
                                   [
                                    ('companyname',mysecfunc),
                                    ('owner', lambda x:"somevalue"),
                                    ('companyemail',myfunc),
                                    ("date",lambda x:datetime.datetime.now()),
                                   ])

loaders = [ContactLoader]