Is there a way to count objects in a GAE data warehouse that satisfy a certain condition? (more than 1000 objects)

Question

I am creating a GAE application that should report on events. The event is of type, and I also need to report the type of event.

For example, let's say there is an event A, B and C. They occur periodically randomly. The user registers and creates a set of objects to which these events can be attributed. When the user returns to check on the status, I need to find out how many events A, B and / or C occurred during a certain period of time, for example, day or month.

The 1000 limit throws a key into the way I usually do it. I don’t need to retrieve all the entities and present them to the user, but I need to show the total for a certain date range. Any suggestions?

I'm a little python / gae noob ...

Answer 1

App Engine is not a relational database, and you won’t be able to quickly count on the fly like this. The best approach is to update the counters while writing, rather than generating them while reading.

When generating counters, there are two general approaches that scale well with the App Engine to minimize the risk of recording:

• Store the account in Memcache or local memory and clean it periodically. This is the simplest solution, but it can be unstable and data loss is possible.
• Use Sharded Counter . This approach is a little more reliable, but more complex. You cannot easily sort by count, but you can also periodically periodically reset to another indexed field of the counter and sort by it.

Answer 2

The data warehouse query results () and offsets for all data warehouse queries are no longer limited to 1000.

Starting from version 1.3.6

Answer 3

My approach would be to have a common model or models for tracking event types, dates, and counts. I am not 100% how you should simulate this, taking into account your requirements.

Then I would postpone the pending tasks to asynchronously update the corresponding aggregate models whenever the user does something that triggers the event.

Nick Johnson Reference work with a deferred library The article contains much more information and presents a structure that may be useful for the kind of aggregation you are talking about.

Answer 4

Will the solution work with cursors (e.g. below)? I personally use this method to count the number of records in a scenario similar to yours, and still have not seen any problems with it (although I work on a schedule, since the constant request of the data warehouse quite burdens the CPU quota).

def count(query): i = 0 while True: result = query.fetch(1000) i = i + len(result) if len(result) < 1000: break cursor = query.cursor() query.with_cursor(cursor) return i 

Answer 5

This post is pretty old, but I would like to provide a useful link. App Engine now offers a built-in API for accessing data warehouse statistics:

For python

 from google.appengine.ext.db import stats global_stat = stats.GlobalStat.all().get() print 'Total bytes stored: %d' % global_stat.bytes print 'Total entities stored: %d' % global_stat.count 

For Java,

 import com.google.appengine.api.datastore.DatastoreService; import com.google.appengine.api.datastore.DatastoreServiceFactory; import com.google.appengine.api.datastore.Entity; import com.google.appengine.api.datastore.Query; // ... DatastoreService datastore = DatastoreServiceFactory.getDatastoreService(); Entity globalStat = datastore.prepare(new Query("__Stat_Total__")).asSingleEntity(); Long totalBytes = (Long) globalStat.getProperty("bytes"); Long totalEntities = (Long) globalStat.getProperty("count"); 

You can also filter the number of objects for a specific type only. Take a look at this link:

https://developers.google.com/appengine/docs/python/datastore/stats https://developers.google.com/appengine/docs/java/datastore/stats

Answer 6

This is very similar to the question I posed on StackOverflow.

How to get excellent value for one of my models in Google App Engine I needed to know how to get excellent values ​​for objects inside my models, and there will be more than 1000 objects for this model.

Hope this helps.