Downloading very large files (8 GB +) - combining WCF and WebClient?

Question

I have some problems creating a WCF service that supports downloading very large files. I read a lot of tutorials on how to set the transferMode attribute in Streamed, increase all messageSize and bufferSize attributes to Int32.MaxValue, and still no luck. (I also return the stream as the body of the message through the MessageBodyMember attribute, and metadata is sent through the headers using the MessageHeader attributes).

If I set all these attributes, I can download the small files in order, but when I try to download 1-2GB files, I just get a 400 error, which makes debugging difficult ...

My service should ideally support a file size of at least 8 GB. Is it even possible with WCF? The various messageSize attributes of the web.config file are apparently limited by Int32.MaxValue, which is equal to a maximum file size of 2 GB.

From my research, I found that it seems to me that I will have to use the WebClient.DownloadFile. Files should only be available for download to users with the necessary rights. With WCF, my upload method can accept a token parameter, which the server can check and return a stream only if the user has permission to download the requested file. This is not like using the WebClient approach. If anyone has any recommendations on how to do this (via WebClient), I would really appreciate it.

Ideally, my WCF service should administer and provide user tokens and somehow attach to each individual file which tokens are currently legal (tokens should be used only once). Then the download should be done through WebClient.

Thanks in advance for any tips.

Answer 1

You can do it in WCF. Many years ago, I built a service that did this (we did not have a web server as part of our configuration). We used WCF streaming:

http://msdn.microsoft.com/en-us/library/ms733742.aspx

The strategy for dealing with large payloads is streaming. Although messages, especially those expressed in XML, are usually considered relatively compact data packets, a message can be a multiple of gigabytes in size and resemble a continuous stream of data more than a data packet. When data is streamed rather than buffered, the sender makes the contents of the message body available to the recipient as a stream, and the message infrastructure continuously transfers data from the sender to the recipient when it becomes available.