502 Bad Gateway: nginx, php5-fpm, 175/209 connect () failed (111: connection refused) when connecting to the upstream

Question

Starting store 5 on a Debian Jessie machine with nginx and php5-fpm , we often get 502 Bad Gateway . This happens mainly in the backend, where longer operations work like creating thumbnails, even if it runs in small pieces of single ajax requests.

The server used with 64 GB of RAM and 16 cores was generally asleep, because there was no real traffic on it. We use it as an intermediate system, unless we have fixed all errors like this.

Error Log:

In the nginx error log, the following lines can be found:

[error] 20524#0: *175 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "POST /backend/MediaManager/createThumbnails HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com", referrer: "http://www.domain.com/backend/" [error] 20524#0: *175 no live upstreams while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "POST /backend/Log/createLog HTTP/1.1", upstream: "fastcgi://php-fpm", host: "www.domain.com", referrer: "http://www.domain.com/backend/" [error] 20524#0: *175 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "GET /backend/login/getLoginStatus?_dc=1457014588680 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com", referrer: "http://www.domain.com/backend/" [error] 20522#0: *209 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "GET /backend/login/getLoginStatus?_dc=1457014618682 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com", referrer: "http://www.domain.com/backend/" 

Perhaps it is noteworthy that the error "* 175 connect first occurs, and then, finally, " * 209 connect .

Configuration files:

I will try to publish only significant lines related to this topic, and will not mark all those lines that are commented out.

PHP-FPM:

/etc/php5-fpm/pool.d/www.conf:

 [www] user = www-data group = www-data listen = /var/run/php5-fpm.sock listen.owner = www-data listen.group = www-data pm.max_children = 5 pm.start_servers = 2 pm.min_spare_servers = 1 pm.max_spare_servers = 3 

Nginx:

/etc/nginx/nginx.conf:

 user www-data; worker_processes auto; pid /run/nginx.pid; events { worker_connections 768; multi_accept on; } http { ## MIME types. include /etc/nginx/mime.types; default_type application/octet-stream; ## Default log and error files. access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; ## Use sendfile() syscall to speed up I/O operations and speed up ## static file serving. sendfile on; ## Handling of IPs in proxied and load balancing situations. # set_real_ip_from 192.168.1.0/24; # set to your proxies ip or range # real_ip_header X-Forwarded-For; ## Timeouts. client_body_timeout 60; client_header_timeout 60; keepalive_timeout 10 10; send_timeout 60; ## Reset lingering timed out connections. Deflect DDoS. reset_timedout_connection on; ## Body size. client_max_body_size 10m; ## TCP options. tcp_nodelay on; ## Optimization of socket handling when using sendfile. tcp_nopush on; ## Compression. gzip on; gzip_buffers 16 8k; gzip_comp_level 1; gzip_http_version 1.1; gzip_min_length 10; gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon application/vnd.ms-fontobject font/opentype application/x-font-ttf; gzip_vary on; gzip_proxied any; # Compression for all requests. gzip_disable "msie6"; ## Hide the Nginx version number. server_tokens off; ## Upstream to abstract backend connection(s) for PHP. upstream php-fpm { server unix:/var/run/php5-fpm.sock; # server 127.0.0.1:9000; ## Create a backend connection cache. keepalive 32; } ## Include additional configs include /etc/nginx/conf.d/*.conf; ## Include all vhosts. include /etc/nginx/sites-enabled/*; } 

/etc/nginx/sites-available/site.conf:

 server { listen 80; listen 443 ssl; server_name xxxxxxxx.com; root /var/www/shopware; ## Access and error logs. access_log /var/log/nginx/xxxxxxxx.com.access.log; error_log /var/log/nginx/xxxxxxxx.com.error.log; ## leaving out lots of shopware/mediafiles-related settings ## .... ## continue: location ~ \.php$ { try_files $uri $uri/ =404; ## NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini fastcgi_split_path_info ^(.+\.php)(/.+)$; ## required for upstream keepalive # disabled due to failed connections #fastcgi_keep_conn on; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param SHOPWARE_ENV $shopware_env if_not_empty; fastcgi_param ENV $shopware_env if_not_empty; # BC for older SW versions fastcgi_buffers 8 16k; fastcgi_buffer_size 32k; client_max_body_size 24M; client_body_buffer_size 128k; ## upstream "php-fpm" must be configured in http context fastcgi_pass php-fpm; } } 

So what's now? Please allow me now if I have to provide additional information on this.

Update

After applying nginx and fpm settings from @peixotorms, errors in nginx logs changed to:

 30 upstream timed out (110: Connection timed out) while reading response header from upstream 

But the problem itself has not been resolved. He just has a different face ...

Answer 1

It may seem strange to you, but your problem is most likely due to the fact that you are using PHP in the socket instead of the tcp port. You will see 502 errors (and others) when you have about 300 simultaneous requests (sometimes less) for php in the socket configuration.

Also your pm.max_children too low if you don't want to limit your server to about 5 concurrent max php requests: http://php.net/manual/en/install.fpm.configuration.php

Configure it this way and these errors should go away:

For your nginx.conf, change the following values:

 worker_processes 4; worker_rlimit_nofile 750000; # handles connection stuff events { worker_connections 50000; multi_accept on; use epoll; } upstream php-fpm { keepalive 30; server 127.0.0.1:9001; } 

Your /etc/php5-fpm/pool.d/www.conf

(Use these settings because you have a lot of RAM or CPU)

 [www] user = www-data group = www-data listen.owner = www-data listen.group = www-data listen.mode = 0660 listen = 127.0.0.1:9001 listen.allowed_clients = 127.0.0.1 listen.backlog = 65000 pm = dynamic pm.max_children = 1024 pm.start_servers = 8 pm.min_spare_servers = 4 pm.max_spare_servers = 16 pm.max_requests = 10000 

Also add this to the location ~ \.php$ { block:

 location ~ \.php$ { try_files $uri $uri/ =404; ## NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini fastcgi_split_path_info ^(.+\.php)(/.+)$; ## required for upstream keepalive # disabled due to failed connections #fastcgi_keep_conn on; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param SHOPWARE_ENV $shopware_env if_not_empty; fastcgi_param ENV $shopware_env if_not_empty; # BC for older SW versions fastcgi_keep_conn on; fastcgi_connect_timeout 20s; fastcgi_send_timeout 60s; fastcgi_read_timeout 60s; fastcgi_pass php-fpm; } 

EDIT:

Change the values ​​below in the /etc/php5/fpm/php.ini file and restart:

 safe_mode = Off output_buffering = Off zlib.output_compression = Off max_execution_time = 900 max_input_time = 900 memory_limit = 2048M post_max_size = 120M file_uploads = On upload_max_filesize = 120M