All geek questions in one place

Handle child lambda failures

We are trying to use lambda for our work with ETL, which is written in Clojure.

Our architecture, the scheduler, initiates the parent lambda, then the parent lambda trigger - 100 child lambda and lambda counter. The lambdas child, after completing his work, will write data to s3. The lambda counter checks the number of files in S3, if it is 100, then it will merge all the files and save them to S3, otherwise it will span the new lambda counter and die.

The whole positive scenario works fine, but if any child fails, the lambda counter will end up in an indefinite loop because there won't be 100 files.

If there is any correct way to reach the child lambda, track it and if it does not work, do you need to restart or retry alone?

Is there a good Clojure lambda structure?

+4
source share
2 answers

AWS has an easy way called AWS Step Features . The Step functions provide a graphical console for organizing and visualizing the components of your application as a series of steps. Define steps using the console or AWS Step APIs, the free Java API, or AWS CloudSormation templates.

The step makes it easy to orchestrate AWS Lambda features. Regardless of function language, it controls all lambdas.

The step is suitable for the following use cases.

  • Execution Functions
  • try/catch/finally
+1

- lambda clojure, , erlang (-) , , , -. " ".

, :

  • :
    • / ( )
  • x100
  • x100

, . , , , , , , aws api. , , , , . , , .

cloudwatch ( riemann), , , .

+2

Source: https://habr.com/ru/post/1686947/

More articles:


All Articles