Bad Gateway 502 error with Apache mod_proxy and Tomcat

Question

We are launching a web application on Tomcat 6 and Apache mod_proxy 2.2.3. Having seen many such errors:

Bad gateway! The proxy received an incorrect response from the upstream server.

The proxy could not process the GET request / the / page.do.

Reason: Error reading from remote server

If you think this is a server error, contact the webmaster.

Error 502

There are many threads in Tomcat, so it is not limited by thread. We are pushing 2,400 users through JMeter against the application. All the boxes are inside our firewall in a fast unloaded network, so there should be no problems with the network.

Anyone have any suggestions on what you can watch or try? We head to tcpdump next.

UPDATE 10/21/08: Still not getting it. See only a very small amount of these under load. The answers below do not give any magic answers ... for now. :)

Answer 1

So, answering my own question. In the end, we determined that we were seeing 502 and 503 errors in the load balancer due to Tomcat timeouts. In the short term, we have increased the waiting time. In the long run, we fixed application problems that cause timeouts in the first place. Why Tomcat timeouts are interpreted as errors 502 and 503 in the load balancer is still a little cryptic.

Answer 2

To add some specific settings, I had a similar setting (with Apache 2.0.63 reverse proxy on Tomcat 5.0.27).

For specific URLs, the Tomcat server may take about 20 minutes to return the page.

As a result, I changed the following parameters in the Apache configuration file to prevent its timings with its proxy operation (with a large overflow factor in case Tomcat took longer to return the page):

Timeout 5400 ProxyTimeout 5400 

---

Some backgound

ProxyTimeout was not enough. Looking at the documentation for Timeout, I assume (I'm not sure) that this is because while Apache is waiting for a response from Tomcat, there is no traffic flow between Apache and the browser (or any other http client), and therefore Apache closes the connection to the browser.

I found that if I left the default Timeout parameter (300 seconds), then if the proxy request for Tomcat took more than 300 seconds to get a response, the browser would display the “502 Proxy Error” page. I believe this message is generated by Apache, knowing that it acts as a reverse proxy before it closes the connection to the browser (this is my real understanding - it may be wrong).

The proxy page says:

Proxy error

The proxy received an invalid response from the upstream server. proxy could not process request GET.

Reason: Error reading from remote server

... which indicates that the ProxyTimeout parameter is too short, while research shows that the Apache Timeout timeout (the timeout between Apache and the client) also affects this.

Answer 3

You can use proxy-initial-not-pooled

See http://httpd.apache.org/docs/2.2/mod/mod_proxy_http.html :

If this variable is set, no pool connection will be reused if the client connection is the initial connection. This avoids the error message “Proxy: error reading line from remote server” caused by a race condition when the internal server closed the connection pool after checking the connection by the proxy server and before the data sent by the proxy server reached the internal server. Keep in mind that setting this variable reduces performance, especially for HTTP / 1.0 clients.

We also had this problem. We fixed this by adding

 SetEnv proxy-nokeepalive 1 SetEnv proxy-initial-not-pooled 1 

and keepAlive on all servers.

mod_proxy_http is suitable for most scenarios, but we run it under heavy load, and we still have some timeout problems that we don’t understand.

But see if the above directive matches your needs.

Answer 4

Example from Apache Conf:

 #Default value is 2 minutes **Timeout 600** ProxyRequests off ProxyPass /app balancer://MyApp stickysession=JSESSIONID lbmethod=bytraffic nofailover=On ProxyPassReverse /app balancer://MyApp ProxyTimeout 600 <Proxy balancer://MyApp> BalancerMember http://node1:8080/ route=node1 retry=1 max=25 timeout=600 ......... </Proxy> 

Answer 5

I assume that you are using mod_proxy_http (or proxy balancer).

Look at your tomcat logs (localhost.log or catalina.log). I suspect you are seeing an exception in your web stack that bubbles up and closes the socket to which the tomcat work object is connected.

Answer 6

you must solve this problem by using the timeout and the proxyTimeout parameter set to 600 seconds. It helped me after the battle for a while.

Answer 7

You can avoid global timeouts or have virtual hosts by specifying proxy server timeouts in the ProxyPass directive as follows:

 ProxyPass /svc http://example.com/svc timeout=600 ProxyPassReverse /svc http://example.com/svc timeout=600 

Pay attention to timeout=600 seconds.

However, this does not always work when you have load balancing. In this case, you should add timeouts in both places (verified in Apache 2.2.31)

Example of load balancing:

 <Proxy "balancer://mycluster"> BalancerMember "http://member1:8080/svc" timeout=600 BalancerMember "http://member2:8080/svc" timeout=600 </Proxy> ProxyPass /svc "balancer://mycluster" timeout=600 ProxyPassReverse /svc "balancer://mycluster" timeout=600 

Note: timeout=600 on ProxyPass not required when Chrome was a client (I don’t know why), but without this timeout on ProxyPass Internet Explorer (11) interrupts the use of the reset connection by the server.

My theory is this:

ProxyPass timeout is used between the client (browser) and Apache.

BalancerMember timeout is used between Apache and the backend.

For those using Tomcat or other support, you can also pay attention to the HTTP connector timeouts.

Answer 8

Most likely, you should increase the Timeout parameter in apache conf (the default value is 120 seconds)

Answer 9

I know this does not answer this question, but I came here because I had the same error with the nodeJS server. I was stuck for a long time until I found a solution. My solution just adds a slash or / at the end of proxyreserve apache.

my old code is:

 ProxyPass / http://192.168.1.1:3001 ProxyPassReverse / http://192.168.1.1:3001 

correct code:

 ProxyPass / http://192.168.1.1:3001/ ProxyPassReverse / http://192.168.1.1:3001/