How to measure the latency of DynamoDB stream propagation?

Question

I am using DynamoDB Streams + Kinesis Client Library (KCL). How can I measure the delay between when the event was created in the stream and when it was processed on the KCL side?

As I know, the KCL MillisBehindLatest KCL MillisBehindLatest specific to Kinesis threads (not DynamoDB threads). The record attribute approximateCreationDateTime has a minute level approximation, which is unacceptable for monitoring on systems with a second second delay.

Could you help with some useful DynamoDB Streams delay tracking metrics?

Answer 1

You can change the recording method in the application so that your application tracks the propagation delay of mutations in the table stream. For example, when creating and updating items, you can always update the timestamp last_updated = attribute. That way, when your creations and updates appear on the stream, you can estimate the propagation delay by subtracting the current time from last_updated in the NEW_IMAGE stream record.

Since exceptions do not have NEW_IMAGE entries in threads, your deletion must be done in two steps:

• logical deletion when you write 'logically_deleted =' timestamp to an element and
• physical deletion when you actually call DeleteItem right after 1.

Then you should use the same math as for creating and updating, only the difference is that you will use OLD_IMAGE when processing deletions, and you will need to subtract at least about 10 ms to take into account the time it takes to execute logical delete (step 1).