How can I start AWS Lambda in response to CouchDB change events using only AWS?

Question

CouchDB provides the _changes , with the ability to use long-lived HTTP connections to provide a stream of change events.

When a change event occurs, I would like to run the AWS Lambda function. The CouchDB continuous change channel seems appropriate, but can this be used as an event source for Lambda?

Typically, you can use the API gateway to call Lambda, but through an HTTP call to the endpoint of the gateway, and not as an HTTP consumer (CouchDB does not seem to provide a mechanism similar to a web host).

This can be solved using follow to listen for changes and manually invoke Lambda, but can this be resolved using AWS?

Answer 1

For the purposes of this answer, I'm going to suggest that “using AWS” excludes other AWS primitives such as EC2 and Fargate. This can be used to solve the problem of continuous streaming.

The short answer is no, there is currently no way in Lambda to keep in touch with a lifetime of more than 300 seconds. Although there are other ways to solve this problem, if you are ready to facilitate the continuous restriction of communication.

The first option is to create a Lambda function that pulls the current change channel and stores the sequence number in the DynamoDB table. You can use the CloudWatch periodic event to trigger this Lambda function each time the CouchDB endpoint is called with the since parameter using the identifier stored in DynamoDB.

The second option is to recursively call your lambda function. Connect to the continuous event feed by passing the since argument and keep the channel open until your Lambda function ends. Before the timeout, take the sequence identifier of the last event and call your Lambda function with this sequence identifier as an input parameter asynchronously, so that your first function ends. There may be a slight delay between calls, but passing the sequence identifier ensures that you do not lose any events.

The third option is to simulate the flow from the second option using Step Functions , which are based on Lambda, but provide a state machine around your lambda calls.

Another option that does not require such coordination between functions, but separates some additional settings, would be to use ECS and Fargate . Fargate manages the entire underlying computing infrastructure for the ECS cluster. You will need to provide container images and ECS configuration (I would recommend CloudFormation), but you will ease the latency limits.