Java UrlConnection throws "Connection reset" exceptions at high load. What for?

Question

I use Java to stream files from Amazon S3 on 64-bit Linux servers (Ubuntu 10).

I use a separate stream for each file, and each file opens an HttpURLConnection , which downloads and processes each file at the same time.

Everything works beautifully until it reaches a certain number of threads (usually around 2-300 parallel threads). At irregular points after this, several (say 10) threads will begin to experience java.net.IOException: Connection reset errors simultaneously .

I am slowing down the download speed and I am doing it below the limit of 250 mbps of the m1.large instance. There is also a slight load on all other aspects of the server (for example, CPU, average load and memory usage are okay).

What could be causing this, or how can I track it?

Answer 1

There is no trivial to guess what might happen, but these are a few tips, maybe some of them may apply in your context:

• You can check your shell (linux bash / zsh or whatever) to see if you raise standard limits limiting the number of file descriptors (but sockets too), man ulimit with a bash shell
• Have you explicitly closed threads in your Java code? non-closing threads can cause such smart problems.
• try google to configure the Linux TCP kernel to try to find out if your ubuntu server has a well-suited stack for such a load context ...

NTN Jerome

Answer 2

They may have an overflow problem on VIP servers due to the number of connections connected to the network reaching the limit. You can reduce the size and see ...

Answer 3

The problem here is mainly in your language. A high load causes an error condition, and an error condition causes an exception. Not the other way around.

Answer 4

One of the common causes of such problems is that an intermediate proxy (firewall, load balancer) reduces what it considers to be an inactive (or too long) HTTP connection. But beyond this general capability, EC2 definitely has more kinks, as others have suggested.

Answer 5

You have probably run out of ephemeral ports . This occurs under load when many short-lived joints quickly open and close. The standard Java HttpURLConnection will not give you the flexibility you need to set the correct socket options. I recommend going to the Apache HttpComponents project and setting options like ...

  ...
 HttpGet httpGet = new HttpGet (uri);
 HttpParams params = new BasicHttpParams ();
 params.setParameter (CoreConnectionPNames.CONNECTION_TIMEOUT, 16 * 1000);  // 16 seconds
 params.setParameter (CoreConnectionPNames.SO_REUSEADDR, true);  // <- teh MOJO!

 DefaultHttpClient httpClient = new DefaultHttpClient (connectionManager, params);
 BasicHttpContext httpContext = new BasicHttpContext ();
 HttpResponse httpResponse = httpClient.execute (httpGet, httpContext);

 StatusLine statusLine = httpResponse.getStatusLine ();
 if (statusLine.getStatusCode ()> = HTTP_STATUS_CODE_300)
 {
 ...

I missed some code like setupManager, but you can get it from your docs.

[Update] You can also add params.setParameter(CoreConnectionPNames.SO_LINGER, 1); to keep the ephemeral ports from dragging on until regeneration.