NGINX: upstream timeout (110: connection timeout) while reading the response header from the upstream

Question

I have Puma working as an application server up and Riak as my db cluster background. When I send a request that converts the card, reduces a piece of data for about 25 thousand users and returns it from the Riak application to the application, I get an error message in the Nginx log:

timeout up (110: connection timeout) while reading the response header upstream

If I request my upstream directly without a nginx proxy, with the same request, I will get the necessary data.

Nginx timeout occurs after entering a proxy.

**nginx.conf** user www-data; worker_processes 2; pid /var/run/nginx.pid; events { worker_connections 4000; } http { sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 10m; proxy_connect_timeout 600s; proxy_send_timeout 600s; proxy_read_timeout 600s; fastcgi_send_timeout 600s; fastcgi_read_timeout 600s; types_hash_max_size 2048; proxy_cache_path /opt/cloud/cache levels=1 keys_zone=cloud:10m; include /etc/nginx/mime.types; default_type application/octet-stream; access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; gzip on; gzip_disable "msie6"; include /etc/nginx/sites-enabled/*.conf; } **virtual host conf** upstream ss_api { server 127.0.0.1:3000 max_fails=0 fail_timeout=600; } server { listen 81; server_name xxxxx.com; # change to match your URL if ($http_x_forwarded_proto != 'https') { return 301 https://$server_name$request_uri; } location / { proxy_pass http://ss_api; # match the name of upstream directive which is defined above proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_cache cloud; proxy_cache_valid 200 302 60m; proxy_cache_valid 404 1m; proxy_cache_bypass $http_authorization; proxy_cache_bypass http://ss_api/account/; add_header X-Cache-Status $upstream_cache_status; } location ~ /\. { deny all; } 

}

Nginx has many timeout directives. I do not know if I am missing something important. Any help would be greatly appreciated ....

Answer 1

You should always refrain from increasing timeouts, I doubt that your response time to the server server is a problem here anyway.

I circumvented this issue by clearing the keep-alive flag of the connection and specifying the http version according to the answer here: https://stackoverflow.com/a/464829/

 server { location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $http_host; # these two lines here proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass http://localhost:5000; } } 

Unfortunately, I can’t explain why this works, and I couldn’t decrypt it from the documents mentioned in the answer related, so if anyone has an explanation, I would be very interested to hear it.

Answer 2

In your case, it helps to optimize the proxy server a bit, or you can use "# timeout"

 location / { # time out settings proxy_connect_timeout 159s; proxy_send_timeout 600; proxy_read_timeout 600; proxy_buffer_size 64k; proxy_buffers 16 32k; proxy_busy_buffers_size 64k; proxy_temp_file_write_size 64k; proxy_pass_header Set-Cookie; proxy_redirect off; proxy_hide_header Vary; proxy_set_header Accept-Encoding ''; proxy_ignore_headers Cache-Control Expires; proxy_set_header Referer $http_referer; proxy_set_header Host $host; proxy_set_header Cookie $http_cookie; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } 

Answer 3

I think this error can occur for various reasons, but it may be specific to the module you are using. For example, I saw this with the uwsgi module, so I had to set "uwsgi_read_timeout".

Answer 4

I would recommend looking at error_logs, in particular on the upstream part, where it shows a specific output that is disabled.

Then based on this, you can configure proxy_read_timeout fastcgi_read_timeout or uwsgi_read_timeout.

Also make sure your configuration is loaded.

Read more here Nginx upstream timeout (why and how to fix)

Answer 5

First find out which upstream is slowing down by consulting the nginx file error log and adjust the read time in my case it was fastCGI

 2017/09/27 13:34:03 [error] 16559#16559: *14381 upstream timed out (110: Connection timed out) while reading response header from upstream, client:xxxxxxxxxxxxxxxxxxxxxxxxx", upstream: "fastcgi://unix:/var/run/php/php5.6-fpm.sock", host: "xxxxxxxxxxxxxxx", referrer: "xxxxxxxxxxxxxxxxxxxx" 

Therefore, I need to configure fastcgi_read_timeout in my server configuration

 ......................... location ~ \.php$ { fastcgi_read_timeout 240; ............ } ................................ 

See: original post

Answer 6

It might be worth a look at http://howtounix.info/howto/110-connection-timed-out-error-in-nginx (it puts proxy_read_timeout in the location block

Answer 7

This is because your upstream is responding too much to the request, and NGINX thinks the upstream has not passed the request, so it responds to an error. Just enable and increase proxy_read_timeout in the location. The same thing happened to me, and I used a 1 hour timeout for an internal application at work:

 proxy_read_timeout 3600; 

At the same time, NGINX will wait an hour until its upstream returns anything.

Answer 8

For our part, he used spdy with a proxy cache. When the cache expires, we get this error until the cache is updated.

Answer 9

I had the same problem and this led to an "every day" error in the rails controller. I don’t know why, but when producing puma it raises an error again and again:

upstream timeout (110: connection timeout) while reading the response header upstream

Probably because Nginx is trying to get data from puma again and again. The funny thing is that the error caused a timeout message, even if I call another action in the controller, so one typo blocks the entire application,

Check the /puma.stderr.log log file to verify that it is.