I have a small application that just polls a server using Jetty v9.2, HttpClient . After a few days, the application will freeze. We initially determined that the thread pool should be increased in size to reduce the impact of performance. This change restored performance in a few days. The lock remains. The reason was isolated from HTTP GET calls (the problem disappears if we comment on the method).
The root cause that appears at the core of the Jetty HttpClient is Connection Management or Flow Management . Usually Jetty HttpClient creates a set of threads for processing HTTP GET (see below), this launch will disappear, as you would expect. After about 40 hours, or the VisualVM JDK runs, it shows at least 9 connection threads that do not go away :
- HttpClient - Scheduler x 1
- HttpClient - SeclectorManager x 4 client selector
- HttpClient x 4
and
Nine or 10 threads. At the next reading, new thread instances are created to transfer the load, and the client continues to work. In addition, the application. It has a clock with a dedicated thread that continues to work after blocking the application, which indicates the JVM, the operating system and the device itself.
Sometimes we see that these “stuck” threads are delayed for an hour before they drop out of the VisualVM thread display. For at least 36 hours we see that the threads remain, and we have not seen how they leave.
After a few days, the software is blocked. The explanation given is the leak of thread instances that have not been flushed. this is an app. runs out of threads and can no longer work. This, of course, stops the HTTP GET, as can be seen from the server logs.
The main HTTP call uses the code below, the HttpClient GET method:
/** * GET * @return null or string returned from server **/ public static String get( final String command ){ String rslt = null; final String reqStr = "http://www.google.com"; // (any url) HttpClient httpClient = new HttpClient(); Request request; ContentResponse response; try { //-- Start HttpClient httpClient.start(); request = httpClient.newRequest( reqStr ); response = request.send(); if( null == response ){ LOG.error( "NULL returned from previous HTTP request."); } else { if( (501 == response.getStatus()) || (502 == response.getStatus()) ){ setNetworkUnavailable(String.format("HTTP Server error: %d", response.getStatus() )); } else { if( 404 == response.getStatus() ){ Util.puts(LOG,"HTTP Server error: 404"); // ignore message since we are talking to an old server } else if( 200 == response.getStatus() ){ rslt = response.getContentAsString(); } else { LOG.error(String.format( " * Response status: \"%03d\".", response.getStatus() )); } setNetworkAvailable(); } } } catch ( InterruptedException iEx ){ LOG.warn( "InterruptException processing: "+reqStr, iEx ); } catch ( Exception ex ){ Throwable cause = eEx.getCause(); if( (cause instanceof NoRouteToHostException) || (cause instanceof EOFException) || (cause instanceof SocketException) && cause.getMessage().startsWith( EX_NETWORK_UNREACHABLE ) ){ setNetworkUnavailable( cause.getMessage() ); } else { LOG.error( "Exception on: "+command, ex ); } } finally { try { httpClient.stop(); } catch ( Exception ex ){ LOG.error( "Exception httpClient.stop(), ServerManager::get()", ex ); } } return rslt; }//get method
This is based on simple examples; there is little information about using HttpClient. Is everything done according to Hoyle?
On different launches, we also see the following Exceptions and log messages:
- [36822522] WARN 2014-Sep-02 02: 46: 28.464> HttpClient @ 2116772232 {STOPPING, 8 <= 0 <= 200, i = 0, q = 0} Could not stop Thread [HttpClient @ 2116772232-729770.5, ]
I wonder if this message refers to one of the stuck threads? Or does this post point to a separate and different issue that we need to study? Also:
- java.util.concurrent.TimeoutException (ExecutionException)
This is apparently a thread timeout exception. Which stream? Is this related to HTTP streams HTTP streams? I think that at least when services catch errors internally, they can at least indicate the location of the error and the stack trace.
There are some obvious questions:
- Is the get () method code, if required, to have no leaks or to leave resources hanging for Jetty's HttpClient code?
- How can we catch the warning: "Could not stop Thread"?
- What is the effect of this error? Is there a way to “break” a stream like this?
- In any case, is it connected with 10 hanging connection threads? There is only one warning message.
- Imagine a hanging thread guarantees an ERROR label, not a warning.
- Is there a way to catch stream errors and errors in general in Jetty HttpClient?
- What attributes are available for HttpClient to configure the service?
- Are there any settings we can use to directly influence thread blocking?
- What attributes are available in the HttpClient environment or context for managing the service?
- Can I restart / restart Jetty HttpClient or just stop it?
- Jetty calls are made only in the GET method shown (albeit with a lot of logging, etc.).
- Does the flow factor use RMI as part of the Jetty HttpClient call?
Another point is that when we “stuck” threads in VisualVM, the Threads panel displays extra Daemon threads, not non-Daemon threads.
By running the code shown in the for loop for about 3 or 4 hours with a break of 250 milliseconds between calls, the HttpClient send () shows a stream leak - it just plays on Linux. Log output does not show WARNings and only two network timeout errors, at least 30 minutes from the stream leak.
Suggestions, comments, improvements and answers are welcome. Thank you for attention.
Related questions :
These questions cover some very similar points.