Removing objects from the application engine

Question

I am going to delete entries from my python application engine server as follows:

try: while True: q = db.GqlQuery("SELECT __key__ FROM SampleData") assert q.count() db.delete(q.fetch(400)) time.sleep(0.2) except Exception, e: self.response.out.write(repr(e)+'\n') pass try: while True: q = db.GqlQuery("SELECT __key__ FROM UserData") assert q.count() db.delete(q.fetch(400)) time.sleep(0.2) except Exception, e: self.response.out.write(repr(e)+'\n') pass 

.. but it just seems ugly, and I continue to suspect that it is not entirely reliable. Is there a better way to do this by deleting records of a certain number of types instead of making one of each of them during loops?

Update. I have one limitation: I run this periodically with a cron job, so I don’t want to do this manually (for example, through the admin console).

Answer 1

Several improvements:

• You do not need to sleep after each batch
• You must use the task queue and be prepared for another task chain if you are not finished by the deadline.
• You should use cursors. If you do not, subsequent queries should skip all the “gravestone” lines that you have already deleted in order to go to existing lines.

If you delete most or all of the views, you can use the mapreduce library instead.

Answer 2

You can use the admin tab of the data warehouse admin console to delete all objects in the view, see here for details:

http://code.google.com/appengine/docs/adminconsole/datastoreadmin.html#Deleting_Entities_in_Bulk

Answer 3

If you want to delete a lot of data, you can use the deferred library provided by Google - there is a cron job that starts a pending task that can delete your objects in batches:

 class DeleteMapper(mapper.Mapper): KIND = MyKindOfObject # Delete all objects of type MyKindOfObject. def map(self, key): todelete = [key] return ([], todelete) 

Most likely, you are similar to the new mapreduce library, however I have no example for you.