Automatically scale ECS cluster to / from zero instances

Question

I have implemented a work observer pattern using SQS and ECS. Job descriptions are transferred to the SQS queue for processing. Job processing is performed on an ECS cluster in an autoscale group that performs ECS Docker Tasks.

Each ECS task performs the following actions:

• Reading a message from the SQS queue
• Performing a task according to data (~ 1 hour)
• Delete message
• Cycle when there are more posts

I would like to reduce the scale of the cluster when there will no longer be work for each instance, eventually to zero instances.

Looking at this similar post , the answers suggest that scaling should be handled outside the ASG in some way. Instances will self-configure, either by explicit self-completion, or by switching the protection of the ASG instance when there are no more messages.

This also does not apply to the case of running several ECS tasks in one instance, since a separate task should not end if other tasks are performed in parallel.

Am I limited to an independent scale and only one task for an instance? Any way to just stop all ECS tasks in the instance? Any other scale alternatives?

Answer 1

You can use CloudWatch Alarms with Actions :

Detect and terminate work instances that are inactive for a certain period of time

Answer 2

I ended up using:

• Scaling policy that adds the same number of instances as pending SQS queues
• Scale in a policy that sets to zero instances when the SQS queue is empty.
• Enabling ASG instance security at the beginning of a batch job and disabling it at the end

This limits me to one batch job per instance, but works well for my scenario.

Answer 3

Another solution to the problem is AWS Batch, announced at the end of 2016.