All geek questions in one place

How to limit consumption from the topic?

Has anyone else solved the following problem?
I have an SNS topic filled with events from S3 , and there is a Lambda function that is subscribed to this topic and when thousands of events fit into this topic, the lambda function is throttled due to exceeding the concurrency limit.
I do not want to request a limit increase for simultaneous executions, but I would reduce the simultaneous consumption from the topic, but I did not find information on how to do this. Thanks.

+5
source share
1 answer

A couple of SNS options:

1) Maximum SNS Receive Rate

Set the maximum SNS reception rate . This will throttle the SNS messages sent to the subscriber, but it may not be a great option if you have so many messages that they will be deleted before they can be processed. From the documentation:

You can set the maximum number of messages per second that Amazon SNS sends to the subscriber endpoint by setting the maximum receive rate setting. Amazon SNS contains messages awaiting delivery up to an hour. Messages stored for more than an hour are discarded.

If you get only thousands of events at a time, setting the maximum receive speed on Lambda is the default limit for running out of 100 at the same time . "Maybe worth a try.

As @kndrk notes, this throttling is only available to HTTP / HTTPS subscribers for the SNS topic. To get around this, you can expose your lambda function through the AWS API gateway and subscribe to this endpoint on the SNS topic, rather than directly to the lambda function.

2) Process from SQS

Sign the SQS queue in the SNS topic and process messages from the queue, and not directly from the sns topic. The SQS ReceiveMessage single instance can only process 10 messages at a time, so it may be easier for you to throttle.

It's also worth noting that you can publish S3 Events directly to AWS Lambda .

+5
source

Source: https://habr.com/ru/post/1259381/

More articles:


All Articles