HttpURLConnections ignore timeouts and never return

Question

We get some unexpected results randomly from some servers when we try to open an InputStream from an HttpURLConnection. It seems that these servers will accept the connection and respond with a “stay alive” header, which will keep Socket open, but will not allow sending data back to the stream.

This scenario makes the multi-threaded crawler attempt a little “complicated” because if some connection gets stuck, the thread executing it will never return ... denying the end of the pool that happens in the controller, thinking that some threads are still working.

Is there a way to read the connection response header to identify the “stay alive” response and not try to open the stream?

Answer 1

I'm not sure what I am missing here, but it seems to me that you just need getHeaderField () ?

Answer 2

Have you tried setting a "read timeout" in addition to a "connection timeout"?

See http://java.sun.com/j2se/1.5.0/docs/api/java/net/URLConnection.html#setReadTimeout%28int%29