How to skip keys in dynamoDB using boto?

Question

I am trying to switch from redis to dynamoDB and sofar, everything works fine! The only thing I still have to find out is the expiration of the key. Currently, I have a data setup with one primary key and no range key:

{ "key" => string, "value" => ["string", "string"], "timestamp" => seconds since epoch } 

What I thought was to do a database scan where the timestamp is less than a certain value and then explicitly delete them. This, however, seems extremely inefficient and will use a ridiculous number of read / write modules for no reason! Also, the expiration will only happen when I run the scan so that they can create.

So, did anyone find a good solution to this problem?

Answer 1

I also use DynamoDB the same way we used Redis.

My suggestion is to write the key in different temporary tables .

For example, let's say that the type of record should last several minutes, not more than an hour, and then you can

• Create a new table every day for this type of record and save the new records in today's table.
• Use the reader to read when you read the records, which means that if you cannot find the record in today's table, you will try to find it in the table yesterday and, if necessary, add it to the table today.
• If you find the entry in any table, check it with a time stamp. There is currently no need to delete expired entries.
• Drop all stale tables in your tasks.

It is easier to maintain and cost effective.

Answer 2

You can make a lazy expiration date and delete it upon request.

For instance:

• Save key "a" with the attribute "expiration", expires after 10 minutes.
• remove after 9 minutes, check the expiration date, return it.
• pick up in 11 minutes. check expiration. since it is smaller than it is now, delete the entry.

This is what memcached did when I looked at the source a few years ago.

You still need to scan to delete all old entries.

You might also consider using Elasticache, which is intended for caching, and not for persistent data storage.

Answer 3

You can use the timestamp as a range key to be indexed and facilitate time-based operations.

Answer 4

It seems Amazon just added support for DynamoDB (starting February 27, 2017). Take a look at the official blog post:

https://aws.amazon.com/blogs/aws/new-manage-dynamodb-items-using-time-to-live-ttl/