I'm having issues with sequential discovery of services using EC2, AWS, Docker, Consul-Template, Consul and NGINX.
I have several services, each of which starts its own instance of EC2. In these cases, I run the following containers (in that order):
- cAdvisor (monitoring)
- node-exporter (monitoring)
- Consul (works in agent mode)
- Registrar
- My service
- Custom container running both nginx and consul-template
The custom container has the following Docker file:
FROM nginx:1.9 #Install Curl RUN apt-get update -qq && apt-get -y install curl #Install Consul Template RUN curl -L https://github.com/hashicorp/consul-template/releases/download/v0.10.0/consul-template_0.10.0_linux_amd64.tar.gz | tar -C /usr/local/bin --strip-components 1 -zxf - #Setup Consul Template Files RUN mkdir /etc/consul-templates COPY ./app.conf.tmpl /etc/consul-templates/app.conf # Remove all other conf files from nginx RUN rm /etc/nginx/conf.d/* #Default Variables ENV CONSUL consul:8500 CMD /usr/sbin/nginx -c /etc/nginx/nginx.conf && consul-template -consul=$CONSUL -template "/etc/consul-templates/app.conf:/etc/nginx/conf.d/app.conf:/usr/sbin/nginx -s reload"
The app.conf file looks like this:
{{range services}} upstream {{.Name}} { least_conn;{{range service .Name}} server {{.Address}}:{{.Port}};{{end}} } {{end}} server { listen 80 default_server; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; location / { proxy_pass http://cart/cart/; } location /cart { proxy_pass http://cart/cart; } {{range services}} location /api/{{.Name}} { proxy_read_timeout 180; proxy_pass http://{{.Name}}/{{.Name}}; } {{end}} }
It seems that everything starts fine, but at some point (which I have not yet identified) after starting up, the consul template seems to return that there are no servers available for a particular service. This means that the upstream section for this service does not contain servers, and I get this in the logs:
2015/12/04 07:09:34 [emerg] 77#77: no servers are inside upstream in /etc/nginx/conf.d/app.conf:336 nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/app.conf:336 2015/12/04 07:09:34 [ERR] (runner) error running command: exit status 1 Consul Template returned errors: 1 error(s) occurred: * exit status 1 2015/12/04 07:09:34 [DEBUG] (logging) setting up logging 2015/12/04 07:09:34 [DEBUG] (logging) config: { "name": "consul-template", "level": "WARN", "syslog": false, "syslog_facility": "LOCAL0" } 2015/12/04 07:09:34 [emerg] 7#7: no servers are inside upstream in /etc/nginx/conf.d/app.conf:336 nginx: [emerg] no servers are inside upstream in /etc/nginx/conf.d/app.conf:336
After that, NGINX will no longer accept requests.
I am sure that I am missing something obvious, but I bound myself with thoughts about the sequence of events, etc. I think NGINX may crash, but since the consul template still works, the Docker container does not restart. I really don't like it if the container itself restarts, or NGINX just restarts.
Can anyone help?