Upload a single file with pieces stored on multiple servers (HTTP)

Question

I am working on a file upload system that will store parts of large files on multiple servers. Thus, the distribution of a 1GB file will look something like this:

Server 1: 0-128MB Server 2: 128MB-256MB Server 2: 256MB-384MB ... etc 

The goal is to allow redundancy (each part will exist on several servers), security (no server has access to the entire file), and cost (bandwidth costs are allocated).

I am curious if anyone has an opinion on how I can “trick” web browsers into downloading different parts from a single link.

What I had in mind was something like:

• The browser is connected to server 1, which provides the size of the contents of the complete file
• After serving 128 MB, server 1 will intentionally close the connection
• Let's hope the browser tries to restart the download by requesting server 1
• Server 1 provides 3XX redirection to server 2
• Browser continues downloading from server 2

I do not know for sure that my example works, since I have not tested it yet. I was curious if there are other solutions that someone might have?

I would like to make the whole process as simple as possible (ideally without work without a simple download). I do not want users to have to use another program (for example: cat'ing files together). I would also like not to use a proxy server, as it will incur additional bandwidth costs.

As far as I know, there is no javascript solution to write the file, if it were, that would be great.

Answer 1

AFAIK is not possible using the HTTP protocol. You may be able to use a custom browser extension, but this will depend on the browser. Another alternative is to create a Java applet that will download the file from different servers. An applet can accept URLs on different servers as parameters.

Answer 2

To save the generated file: fooobar.com/questions/26832 / ...

This solution stores the file in memory, so it will not work with very large files.

You can load partial files into a JS variable using JSONP. It will also allow you to circumvent a policy of the same origin.

Answer 3

The Javascripts security model will allow you to access data from the same source where Javascript came from, i.e. not to multiple servers.

If you are going to have file bits on multiple servers, you will need a user to load the web page, receive the bits, and then combine the bits in the correct order. If you manage to get all your users to do this (right), you're better than me.

Answer 4

This can be done in modern browsers using the HTTP standard.

You can use XHR2 with CORS to load fragments of a file as an ArrayBuffer , and then merge them using the Blob constructor and use createObjectURL to send the merged file to the user.

However, I suspect that browsers will store these objects in RAM, so it is probably a bad idea to use it for large files.