Background Intelligent Transfer Service and Amazon S3

Question

I am using SharpBITS to download a file from AmazonS3.

> // Create new download job. BitsJob > job = this._bitsManager.CreateJob(jobName, JobType.Download); > // Add file to job. > job.AddFile(downloadFile.RemoteUrl, downloadFile.LocalDestination); > // Resume > job.Resume(); 

It works for files that do not require authentication. However, as soon as I add an authentication request line to request an AmazonS3 file, the response from the server is an HTTP 403 state -automatized. Url works in a browser.

Here is the HTTP BIT service request:

 HEAD /mybucket/6a66aeba-0acf-11df-aff6-7d44dc82f95a-000001/5809b987-0f65-11df-9942-f2c504c2c389/v10/summary.doc?AWSAccessKeyId=AAAAZ5SQ76RPQQAAAAA&Expires=1265489615&Signature=VboaRsOCMWWO7VparK3Z0SWE%2FiQ%3D HTTP/1.1 Accept: */* Accept-Encoding: identity User-Agent: Microsoft BITS/7.5 Connection: Keep-Alive Host: s3.amazonaws.com 

The only difference between the browser is the type of request. Firefox makes a GET request, and BITS makes a HEAD request. Are there any problems with Amazon S3 HEAD requests and query string authentication?

Regards, Blaz

Answer 1

You are probably right that a proxy server is the only way around this. BITS uses the HEAD request to get the length of the content and decides whether it wants to block the download of the file. Then a GET request is executed to actually retrieve the file — sometimes in general if the file is small enough, otherwise with range headers.

If you can use a proxy server or some other trick to give it some kind of response to a HEAD request, it should peel off. Even if the HEAD request is faked with dummy content length, BITS will go to GET. You can see duplicate GET requests in this case, because if the first GET request returns the length of the content longer than the original HEAD request, BITS can decide "oh shit, I will do it best."

Given this, I'm a little surprised that he is not smart enough to recover from a 403 error in a HEAD request and still move on to GET. What is the actual work behavior? Have you tried watching it using bitadmin / monitor? If the task is in a state of transient error, it can do this for about 20 minutes, and then eventually recover.

Answer 2

Before starting the download, BITS sends an HTTP HEAD request to the server to find out the size of the deleted file, timestamp, etc. This is especially important for BranchCache-based BITS transfers and is the reason that server-side HTTP HEAD support is specified as an HTTP requirement for BITS downloads .

In this case, BITS bypasses the HTTP HEAD request phase and immediately issues an HTTP GET request if one of the following conditions is true:

• The BITS task is configured with the BITS_JOB_PROPERTY_DYNAMIC_CONTENT flag.
• BranchCache is disabled And the BITS job contains one file.

A workaround (1) is most appropriate since it does not affect other BITS transmissions in the system.

For workaround (2), BranchCache can be disabled using the BITS DisableBranchCache Group Policy. You will need to run gpupdate from an elevated command prompt after making changes to Group Policy, or it will take about 90 minutes for the changes to take effect.