How to automate the AWS EC2 group scale in relatively complex infrastructures?

Question

I am working to migrate our servers to Amazon Cloud, for reasons of autoscaling, cost, services, etc., obviously.

Until now, I am experimenting a lot and trying to dive into the fully functional documentation, but without previous experience I have many questions.

The provided infrastructure is as follows:

+-----+ | ELB | +--+--+ | +--------------------|--------------------+ | Auto-Scaling Group | |--------------------|--------------------| | | | | +---------+ | +---------+ | | | varnish |<------+------>| varnish | | | +----+----+ +---------+ | | | | | +-----------------------------------------+ | | | | | +------------+ | +---->|Internal ELB|<-----+ +------+-----+ | +-----------------------------------------+ | Auto-Scaling Group | |-----------------------------------------| | +---------+ | +---------+ | | | Apache |<------+------>| Apache | | | +----+----+ +----+----+ | | | | | +-----------------------------------------+ | +-----+ | +-------->| RDS |<--------+ +-----+ 

In words, I will have Elastic LoadBalancer, which will send traffic to Varnish instances, which, in turn, will send traffic to the internal Elastic LoadBalancer, which will send traffic to Apache interfaces.

Currently, I have discovered AWS tools, such as CloudFormation , which seems to be able to load the instance specified by the template, seems to be great, but it seems to only load.

Having little experience with Puppet (and considering AWS's recommendation on this), I fell in love with the Puppet thing, which is a great tool.

My idea, which may be unviable or realistic, is to create a “puppet Node stack” using CloudFormation templates that will customize the instance as needed and connect the doll master layout to it.

Once my stack is ready, I wonder how to set up / create an Auto-Scaling group for Varnish and Apache instances.

It seems that CFN has the resources to configure groups and auto-scaling policies, so I think I could create two different templates for each.

But will the AS function work through the CFN service and then do all the init actions (and do user-data )?

I also read here and there that Puppet can use EC2 tags, maybe a common stack template with corresponding tags (e.g. roles) could do the trick?

Is this architecture realistic and sustainable? Do you have any feedback?

Thanks for your advice.

Answer 1

Automatic scaling creates new nodes based on the launch configuration. Thus, you will have two separate auto-scaling groups and two separate startup configurations. i.e

 "VarnishScalingGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "LaunchConfigurationName" : {"Ref" : "VarnishLaunchConfiguration" }, "LoadBalancerNames" : {"Ref" : "ELB"}, ... } }, "VarnishLaunchConfiguration" : { "Type" : "AWS::AutoScaling::LaunchConfiguration", "Properties" : { ... "UserData" : { .... }, "MetaData" : { ... } }, "ApacheScalingGroup" : { "Type" : "AWS::AutoScaling::AutoScalingGroup", "Properties" : { "LaunchConfigurationName" : {"Ref" : "ApacheLaunchConfiguration" }, "LoadBalancerNames" : {"Ref" : "InternalELB"}, ... } }, "ApacheLaunchConfiguration" : { "Type" : "AWS::AutoScaling::LaunchConfiguration", "Properties" : { ... "UserData" : { .... }, "MetaData" : { ... } } 

Another thing you want to add is separate scaling policies for each scaling group and corresponding CloudWatch metrics to match.

CloudFormation can also trigger stack updates. If you kick cfn-hup as part of the user data, then it periodically (you decide) checks for changes in the metadata of the stack, and then does whatever you prefer. I tend to run another version of cfn-init - which will parse and update any metadata.

The key point is if you go down the cfn-hup path, it will not execute user data again if the CloudFormation stack does not require resetting and creating new instances.

Another point, if you want updates for LaunchConfiguration to be deployed, you need to make sure that LaunchConfiguration also has the UpdatePolicy app.

Answer 2

Instead of having a “Puppet Node Stack,” you might consider pre-creating your AMI with a tool such as a packer ( http://www.packer.io/ ) that can provide a puppet machine and create an AMI. Then add the prepared AMI to the cloud information template.

According to Peter H., cloudformation can handle updates to your stack. Therefore, when you make changes to your puppet installation, you can create a new AMI and update the launch configuration in cloud form. Autoscaling will start using the new AMI to automatically delete new instances.

Taking a puppet out of cloud information gives you a separation of problems between infrastructure and server configuration.

Scaling will be faster with preconfigured AMIs that already have Apache / Varnish settings.

There are also benefits to a puppet installation workshop. i.e. Decentralized, without a puppeteer as a point of failure, etc. See https://serverfault.com/questions/408261/pros-and-cons-of-a-decentralized-puppet-architecture