Redis: excluding values ​​from a sorted set based on the value of the hash field

Question

I was wondering if anyone could give some suggestions on how to make orderly creation of a set more efficient?

I am working on a project in which ranking data is computed hourly and stored in a database. Data can be filtered by member gender, country, etc. There are about 2 million lines that need to be processed, and this takes a lot of time.

We want to move on to a more realistic approach when data is saved / sorted / filtered in Redis and daily clean recovery.

In my prototype, I create a sorted set for each possible combination of filters, for example: leaderboard.au.male, leaderboard.au.female, etc. I wrote this process, but as soon as you process each case, it means that 118 sorted sets are created there.

Ideally, I would like to have one sorted set of ranks and hash sets for each member containing their name, gender and country. Then using Redis only returns sorted set values ​​based on custom filters. (e.g. get ratings only for men from Australia).

Is it possible to do this initially in Redis?

Answer 1

I suggest you keep a set with a rating for all participants:

leaderboard = { id1: score1, id2: score2, ... } 

And a set for each type (gender, country, etc.):

 members.male = { id1, id2, ... } members.au = { id2, id3, ... } 

Then you do ZINTERSTORE :

 zinterstore leaderboard.male 2 leaderboard members.male 

Or, to get the leaderboard for male members:

 zinterstore leaderboard.au.male 3 leaderboard members.male members.au 

You can control how the count of results for a sorted set should be calculated using WEIGHTS and AGGREGATE .

If you do not want to save result sets for a long time, you could EXPIRE them and only build a new set if it does not exist.