The upstream server is temporarily down when connecting to the upstream

Question

I have 2 harbor servers running under nginx server (acting as load balancing and reverse proxy), namely harbor.

nginx config load balance:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    upstream harbor {
        ip_hash;
        server 10.57.18.120;
        server 10.57.18.236;
    }

    server{
        listen 80;
        location / {
            proxy_pass http://harbor;
        }
    }
}

nginx config in the harbor:

worker_processes auto;

events {
  worker_connections 1024;
  use epoll;
  multi_accept on;
}

http {
  tcp_nodelay on;

  # this is necessary for us to be able to disable request buffering in all cases
  proxy_http_version 1.1;


  upstream registry {
    server registry:5000;
  }

  upstream ui {
    server ui:80;
  }


  server {
    listen 80;

    # disable any limits to avoid HTTP 413 for large image uploads
    client_max_body_size 0;

    location / {
      proxy_pass http://ui/;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
      # proxy_set_header X-Forwarded-Proto $scheme;

      proxy_buffering off;
      proxy_request_buffering off;
    }

    location /v1/ {
      return 404;
    }

    location /v2/ {
      proxy_pass http://registry/v2/;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
      # proxy_set_header X-Forwarded-Proto $scheme;

      proxy_buffering off;
      proxy_request_buffering off;

    }

    location /service/ {
      proxy_pass http://ui/service/;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
      # proxy_set_header X-Forwarded-Proto $scheme;

      proxy_buffering off;
      proxy_request_buffering off;
    }
  }
}

When both upstream servers are working, everything is fine, but if one upstream is down, nginx cannot forward requests to the server. Here are the logs:

2016/11/17 09:05:28 [error] 6#6: *1 connect() failed (113: No route to host) while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://10.57.18.236:80/", host: "10.57.18.236:2000"
2016/11/17 09:05:28 [warn] 6#6: *1 upstream server temporarily disabled while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://10.57.18.236:80/", host: "10.57.18.236:2000"
2016/11/17 09:05:28 [error] 6#6: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://10.57.18.120:80/", host: "10.57.18.236:2000"
2016/11/17 09:05:28 [warn] 6#6: *1 upstream server temporarily disabled while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://10.57.18.120:80/", host: "10.57.18.236:2000"
10.57.2.138 - - [17/Nov/2016:09:05:28 +0000] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" "-"
2016/11/17 09:05:28 [error] 6#6: *1 no live upstreams while connecting to upstream, client: 10.57.2.138, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://apps/favicon.ico", host: "10.57.18.236:2000", referrer: "http://10.57.18.236:2000/"
10.57.2.138 - - [17/Nov/2016:09:05:28 +0000] "GET /favicon.ico HTTP/1.1" 502 575 "http://10.57.18.236:2000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" "-"
2016/11/17 09:05:34 [error] 6#6: *7 no live upstreams while connecting to upstream, client: 10.57.2.138, server: , request: "GET / HTTP/1.1", upstream: "http://apps/", host: "10.57.18.236:2000"
10.57.2.138 - - [17/Nov/2016:09:05:34 +0000] "GET / HTTP/1.1" 502 173 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/601.6.17 (KHTML, like Gecko) Version/9.1.1 Safari/601.6.17" "-"

It shows "upstream server temporarily disabled while connecting to upstream"and "no live upstreams while connecting to upstream"when the upward flow down, but up techeniyu2 still up.

But I still get "502 Bad Gateway"if I use domainUrl. Currently browsing in upstream2 (via IP) in the browser is fine.

I tried to add the same problem "proxy_next_upstream"in http, on the server, in the block location /.