All geek questions in one place

Using Custom AMI (with s3cmd) in Datapipeline

How to install s3cmd on AMI, which is used in the pipeline? This should be a fairly simple task, but I cannot do it: Here is what I tried:

  • Pipeline started without the option Image-id => Everything works fine
  • Moved to EC2 and created an image of the running instance, to make sure that all the necessary materials that will be executed in the pipeline will be installed on my regular AMI
  • Started this AMI manually as an instance.
  • Ssh'd into the machine and installed S3cmd
  • Another machine image was created, this time with s3cmd installed
  • Shutdown Instance
  • The pipeline is launched again, this time with the newly created AMI when the image ID and S3cmd are set.

Now the resource runs “RUNNING”, but my activity (ShellCommandActivity) is stuck in WAITING_FOR_RUNNER state, and the script never runs.

What do I need to do to create a pipeline with a custom image? Or is there an even simpler way to use s3cmd in the pipeline?

Thanks!

+3
source share
1 answer

I realized this now, using a “clean” Amazon Linux AMI (for example, on the market) and installing S3cmd, instead of creating an AMI from a working Pipeline resource. I saw that this AMI has a different kernel version, so this could be a problem.

+2
source

Source: https://habr.com/ru/post/1493420/

More articles:


All Articles