How to deploy a Python app on Amazon's Elastic Beanstalk from Jenkins?

Question

I am trying to deploy to Amazon Elastic Beanstalk programmatically from Jenkins work. On my development machine, it is as simple as:

eb deploy $(AWS_ELASTIC_BEANSTALK_ENVIRONMENT) 

In Jenkins, this should be as simple as setting up the following command as a build command:

 virtualenv env && source env/bin/activate && pip install awsebcli mkdir -p .elasticbeanstalk cat << EOF > .elasticbeanstalk/config.yml branch-defaults: master: environment: myenv global: application_name: myapp default_ec2_keyname: null default_platform: 64bit Amazon Linux 2014.09 v1.0.9 running Python 2.7 default_region: us-west-2 profile: eb-cli sc: git EOF eb deploy myenv 

However, this does not work with the following trace:

 [EnvInject] - Loading node environment variables. Building remotely on standard Amazon Linux 2014.09 AMI (i-d39710df) (x) in workspace /media/ephemeral0/jenkins/workspace/My_Job Fetching changes from the remote Git repository Fetching upstream changes from [e-mail] :myapp.git Checking out Revision be45db94111f9ab49fe8031eb583307d2fb9921c (origin/master) [My_Job] $ /bin/sh -xe /tmp/hudson8633484437962332339.sh + virtualenv env New python executable in env/bin/python2.7 Not overwriting existing python script env/bin/python (you must use env/bin/python2.7) Installing Setuptools..................................................................................................................done. Installing Pip....................................................................................................................................done. + source env/bin/activate ++ deactivate nondestructive ++ unset pydoc ++ '[' -n '' ']' ++ '[' -n '' ']' ++ '[' -n /bin/sh -o -n '' ']' ++ hash -r ++ '[' -n '' ']' ++ unset VIRTUAL_ENV ++ '[' '!' nondestructive = nondestructive ']' ++ VIRTUAL_ENV=/media/ephemeral0/jenkins/workspace/My_Job/env ++ export VIRTUAL_ENV ++ _OLD_VIRTUAL_PATH=/usr/local/bin:/bin:/usr/bin:/opt/aws/bin ++ PATH=/media/ephemeral0/jenkins/workspace/My_Job/env/bin:/usr/local/bin:/bin:/usr/bin:/opt/aws/bin ++ export PATH ++ '[' -n '' ']' ++ '[' -z '' ']' ++ _OLD_VIRTUAL_PS1= ++ '[' x '!=' x ']' +++ basename /media/ephemeral0/jenkins/workspace/My_Job/env ++ '[' env = __ ']' +++ basename /media/ephemeral0/jenkins/workspace/My_Job/env ++ PS1='(env)' ++ export PS1 ++ alias 'pydoc=python -m pydoc' ++ '[' -n /bin/sh -o -n '' ']' ++ hash -r + pip install awsebcli Requirement already satisfied (use --upgrade to upgrade): awsebcli in ./env/lib/python2.7/site-packages Downloading/unpacking setuptools>=7.0 (from awsebcli) Running setup.py egg_info for package setuptools Requirement already satisfied (use --upgrade to upgrade): pyyaml>=3.11 in ./env/lib/python2.7/site-packages (from awsebcli) Requirement already satisfied (use --upgrade to upgrade): six==1.8.0 in ./env/lib/python2.7/site-packages (from awsebcli) Requirement already satisfied (use --upgrade to upgrade): cement==2.4 in ./env/lib/python2.7/site-packages (from awsebcli) Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.2 in ./env/lib/python2.7/site-packages (from awsebcli) Requirement already satisfied (use --upgrade to upgrade): jmespath>=0.4.1 in ./env/lib/python2.7/site-packages (from awsebcli) Installing collected packages: setuptools Found existing installation: setuptools 0.9.7 Uninstalling setuptools: Successfully uninstalled setuptools Running setup.py install for setuptools Installing easy_install script to /media/ephemeral0/jenkins/workspace/My_Job/env/bin Installing easy_install-2.7 script to /media/ephemeral0/jenkins/workspace/My_Job/env/bin Successfully installed setuptools Cleaning up... + mkdir -p .elasticbeanstalk + cat + cat .elasticbeanstalk/config.yml branch-defaults: master: environment: myenv global: application_name: myapp default_ec2_keyname: null default_platform: 64bit Amazon Linux 2014.09 v1.0.9 running Python 2.7 default_region: us-west-2 profile: eb-cli sc: git + eb deploy myenv ERROR: The config profile (eb-cli) could not be found Build step 'Execute shell' marked build as failure Finished: FAILURE 

It is not clear why this has been happening since I run the above local copy of my project, it works fine.

The error message doesn't seem to help much. It is unclear why eb-cli can not be found on the Jenkins machine.

So, to summarize my question again: how do I deploy to Jenkins Amazon Elastic Beanstalk? Is the above approach correct, but with errors in detail? Or is there some simpler way?

Answer 1

I resolved this using ssh ing to the Jenkins machine by running eb init and then comparing the generated .elasticbeanstalk/config.yml with the one that was used in this document, which I used above. These two options were different due to different security profiles on my development machine compared to the Jenkins machine.

We can rewrite this script to be more resistant to various config.yaml files as follows:

 virtualenv env && source env/bin/activate && pip install awsebcli echo "1" | eb init myapp --region us-west-2 && eb use myenv && eb deploy myenv 

Note. We use echo "1" | eb init myapp --region us-west-2 echo "1" | eb init myapp --region us-west-2 to select the default environment, since eb init does not accept the environment as a positional argument, and then use eb use myenv to select the desired environment.

Answer 2

To fix the config profile (eb-cli) could not be found error, release the credentials that you use to deploy to EB to ~/.aws/config for your jenkins user on your jenkins machine. If you built your deployment on a local computer, you can pull the file directly from ~/.aws/config locally. It will look like this:

 [profile eb-cli] aws_access_key_id = (for your IAM user) aws_secret_access_key = (for your IAM user) 

Answer 3

We faced this problem not in blue, it may or may not be relevant, but I wanted to get it there for everyone who might encounter it again. Probably eb-cli has changed a bit and does not allow credentials to be set worldwide.

Our JenkinsFile as follows

  withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'iam-creds']]) { sh "pip install awsebcli --upgrade --user" sh "~/.local/bin/eb use my-application" sh "~/.local/bin/eb deploy --verbose" } 

and our config.yml looks like this:

  global: application_name: my-application default_ec2_keyname: application-keyname ... profile: eb-cli sc: git workspace_type: Application 

deleting the profile key solves the problem ... However, this prohibits deployment from the local computer (unless there is a way to use global characters)