How do you handle Amazon Kinesis Record duplicates?

Question

According to Amazon Kinesis Streams documentation, a record can be delivered multiple times.

The only way to check each record only once is to temporarily save them in a database that supports integrity checks (for example, DynamoDB, Elasticache or MySQL / PostgreSQL) or just a RecordId breakpoint for each Kinesis fragment.

Do you know a better / more efficient way to handle duplicates?

Answer 1

. , , , MD5 , - , , .

-, , , , , , Kinesis, .

, DynamoDB MD5 . , - , , DynamoDB , , ( ). , . DynamoDB, .

DynamoDB - Time To Live, , , . DynamoDB, , ElastiCache, ElastiCache ( , Memcached) - , , , .

Answer 2

, , " ". , , (- ReadTimeout ..). . , .

" " . , AWS SQS : https://aws.amazon.com/about-aws/whats-new/2016/11/amazon-sqs-introduces-fifo-queues-with-exactly-once-processing-and-lower-prices-for-standard-queues/. , SQS , Kinesis.

, , ( ). , (, , ), , ( ).

. :

• : http://www.cloudcomputingpatterns.org/at_least_once_delivery/
• - : http://www.cloudcomputingpatterns.org/exactly_once_delivery/
• Idempotent Processor: http://www.cloudcomputingpatterns.org/idempotent_processor/