Does Apache HttpClient 4.x behave strangely when uploading large files?

Question

I develop and test a small direct client-server application using java (and scala).

The server is based on com.sun.net.httpserver.HttpServer and allows you to upload files through the RESTful basic interface using the POST and PUT operations. The download operation is limited by the "Digest Authentication" , which we implemented ourselves, verified and works in browsers, curls and Apache HttpClient .

download client terminates Apache HttpClient 4.1.2 and performs PUT operations via http to download files. The file content type is indicated as application/xml in the header, and only one file is downloaded at a time.

When downloading files of different sizes, you can observe strange behavior:

• Files smaller than or equal to 1.076.006 bytes are downloaded successfully .
• Files with sizes greater than or equal to 1.122.158 bytes failed with java.net.SocketException: Broken pipe .

(The exact critical size is not known since I created files of different sizes manually to get closer to the maximum working size)

The cause of the faulty channel is that the client somehow ignored downloading www-authenticate -response files of this size, as described in the server logs. Ignore means that it simply sends a few (4) messages that do not contain an authentication header. But smaller files work well, and the client sends the authentication request with the appropriate response request correctly right after www-authenticate answer is as it should be.

Downloading works in curl with files of all sizes, so there is no problem.

So, at this stage you can say: "There is an error in your client." Well, I hope so, but I also tried java RESTclient (also wrapping apache httpclient) and it has exactly the same behavior!

We tried it using this client over the Internet, and also in the same way as described. So right now, I just hope that I skipped to install something important in Apache HttpClient , which leads to this erroneous behavior, and the open source RESTclient developer also skipped it ... any ideas what this could be , it would be great

Answer 1

Most likely, this is a combination of several factors that lead to this situation.

(1) Most likely, your client does not use a wait-continue handshake when sending a large request object with a request that does not include an authentication header.

(2) The server detects that the request does not live up to expectations, and instead of reading and discarding the body of the full request, it responds early to status 401 and closes the connection at its end. In my opinion, this is a violation of the HTTP protocol by the server.

(3) Although some HTTP agents may handle early responses, Apache HttpClient cannot because of the restriction of Java I / O blocking (the thread can either read or write from a blocking socket, but not for both).

There are several ways to solve this problem, the “wait-continue” handshake is the simplest and most natural. Alternatively, you can run a simple HEAD or GET request to force HTTP authentication before executing a large POST or PUT request. HttpClient is able to reuse authentication data for subsequent requests in the same logical HTTP session.