XMLHttpRequest and Chrome Developer Tools do not say the same

Question

I upload a 50 MB file into 5 MB chunks using XMLHttpRequest and Range header. Everything works fine except when I downloaded the last snippet.

Here is a screenshot of the request and response for the first fragment. Please note that the content length is 1024 * 1024 * 5 (5 MB). Also note that the server responds correctly with the first 5 MB, and in the Content-Range header sets the size of the entire file (after / ) correctly:

When I copy the response body to a text editor (Sublime), I get only 5,242,736 characters instead of the expected 5,242,880, as indicated by the Content-Length :

Why are 144 characters left? This is true for every piece that loads, although the exact difference is slightly different.

However, which is especially strange, this is the last piece. The server responds with the last ~ 2.9 MB of the file (instead of as much as 5 MB) and, apparently, correctly indicates this in the answer:

Please note that I am requesting the following 5 MB (even if it goes beyond the total file size). No biggie, the server responds with the last part of the file, and the headers indicate the actual range of bytes.

But is it really?

When I call xhr.getResponseHeader("Content-Length") with Javascript, I see a different story in Chrome:

The XMLHttpRequest object tells me that another 5 MB has been downloaded outside the file. I don’t understand something about the xhr object?

What's even weirder is that it works in Firefox 30, as expected:

Thus, between xhr.responseText.length does not match Content-Length , and these headers are not consistent between the xhr object and the network tools, I do not know what to do to fix this.

What causes these discrepancies?

Update: I confirmed that the server itself sends the request correctly, despite exceeding the Range header in the request for the last fragment. This is the result of a raw HTTP request, thanks to the nice 'ol telnet :

 HTTP/1.1 206 Partial Content Server: nginx/1.4.5 Date: Mon, 14 Jul 2014 21:50:06 GMT Content-Type: application/octet-stream Content-Length: 2987360 Last-Modified: Sun, 13 Jul 2014 22:05:10 GMT Connection: keep-alive ETag: "53c30296-2fd9560" Content-Range: bytes 47185920-50173279/50173280 

So, it looks like Chrome is not working properly. Should it be logged as an error? Where?

Answer 1

The main problem is that you are reading binary data as text. Please note that the server responds with Content-Type: application/octet-stream , which does not explicitly indicate the encoding - in this case, the browser usually assumes that the data is encoded in UTF-8. Although the length will basically not change (bytes with values ​​from 0 to 127 are interpreted as a single character in UTF-8, and bytes with higher values ​​are usually replaced by a replacement character), your binary certainly contains some valid multi-byte UTF- 8, and they will be combined into one character. This explains why responseText.length does not match the number of bytes received from the server.

Now you could, of course, force a specific encoding using the request.overrideMimeType() method, then according to ISO 8859-1 it would be especially important because the first 256 Unicode code points are identical to ISO 8859-1:

 request.overrideMimeType("application/octet-stream; charset=iso-8859-1"); 

This should ensure that one byte will always be interpreted as one character. However, the best approach would be to store the server response in ArrayBuffer , which is explicitly designed to handle binary data.

 var request = new XMLHttpRequest(); request.open(...); request.responseType = "arraybuffer"; request.send(); ... var array = new Uint8Array(request.response); alert("First byte has value " + array[0]); alert("Array length is " + array.length); 

According to MDN , responseType = "arraybuffer" has been supported since Chrome 10, Firefox 6, and Internet Explorer 10. See also: Typed arrays .

Side-note: Firefox also supports responseType = "moz-chunked-text" and responseType = "moz-chunked-arraybuffer" , starting with Firefox 9, which allow you to receive data in chunks without resorting to range requests. Chrome doesn't seem to have plans to implement it; instead, they work when implementing the Streams API .

Change I have not been able to reproduce your problem with Chrome lying in front of you in the response headers, at least without your code. However, the responsible code must be this function in partial_data.cc :

 // We are making multiple requests to complete the range requested by the user. // Just assume that everything is fine and say that we are returning what was // requested. void PartialData::FixResponseHeaders(HttpResponseHeaders* headers, bool success) { if (truncated_) return; if (byte_range_.IsValid() && success) { headers->UpdateWithNewRange(byte_range_, resource_size_, !sparse_entry_); return; } 

This code will remove the Content-Length and Content-Range headers returned by the server and replace them with those generated from your request parameters. Given that I cannot reproduce the problem myself, the following are just guesses:

• This code path seems to be used only for requests that can be satisfied from the cache, so I think that everything will work correctly if you clear your cache.
• resource_size_ variable must have the wrong value in your case, greater than the actual size of the requested file. This variable is determined from the Content-Range header in the requested first fragment, perhaps you have a cache server that points to a larger file.