Counting / querying events with multiple keys and dates in Redis

Question

I am trying to figure out how to process my data structure in Redis. What I'm trying to accomplish is how I can count events with two parameters and then query Redis for this data by date. Here is an example: events come in with two different parameters, they can be called the site and type of event, as well as with the time this event occurred. From there, I will need to query Redis for how many events have occurred over a period of dates, grouped by site type and type.

Here is a brief example of a dataset:

Oct 3, 2012: site A / event A site A / event A Site B / event A Oct 4, 2012: site B / event B site A / event A Site B / event A 

... etc.

In my request, I would like to know the total number of events over a period that will last five weeks. In the above example, it would be something like this:

  site A / event A ==> 3 events site B / event A ==> 2 events site B / event B ==> 1 event 

I examined the use of Redis sort function, hashes, etc. Sorted Set seems to be the best way to do this, but querying data with the Redis "ZUNIONSTORE" command doesn’t seem that big, because these events will take five weeks. This gives at least 35 arguments to the ZUNIONSTORE command.

Any hints, thoughts, ideas, etc.

Many thanks for your help.

Answer 1

Unlike typical RDBMS or MongoDB, Redis does not have a rich query language that you can use. In these stores, you accumulate raw data in the warehouse, and then you can use the query to calculate statistics. Redis is not adapted to this model.

With Redis, you need to calculate your statistics on the fly and store them directly instead of raw data.

For example, suppose we are only interested in statistics for several weeks, I would structure the data as follows:

• because all criteria are discrete, simple hash objects can be used instead of zsets

• one hash object per week

• in each hash object, one counter per pair of sites, an event. Optionally, one counter per site and / or one counter per event.

So, when the event occurs, I would execute the following commands for Redis:

 hincrby W32 site_A:event_A 1 hincrby W32 site_A:* 1 hincrby W32 *:event_A 1 

Note: there is no need to initialize these counters. HINCRBY will create them (and the hash object) if they do not exist.

To get statistics for one week:

 hgetall W32 

In statistics, you have counters for each site / event, only for each site.

To get statistics for several weeks, run the following commands:

 hgetall W32 hgetall W33 hgetall W34 hgetall W35 hgetall W36 

and perform aggregation on the client side (quite simple if the language supports associative arrays, such as map, dictionary, etc.).