Stream large file to web service

Question

We have a web service (asmx) that takes an array byte buffer and passes it to an external file service for storage and archiving. It is called by the Windows service, which generates fairly small files (3-5M), so the creation of an array of bytes, the size and supply of which as a parameter to call the web service, still worked.

I was instructed to add a new job to the file queue that could potentially generate s> 70M files. Obviously, byte[] this size will overload system memory, so I decided to use solutions. The web service has an internal method that takes FileStream as a parameter instead of byte[] , so I tried using the FileStream method as WebMethod. I updated the links in the Windows service, but the strange thing happens: the FileStream parameter has an attached namespace specifier (something like CallingWindowsService.ExternalWebServiceFileManager.FileStream ), and this specified object does not accept the file name as a constructor, so I cannot open a specific file.

I am completely at sea, how to approach this. Has anyone else done this - streaming large amounts of data for a web service - and if so, what is the best method? My web service will require either byte[] or FileStream .

Looking through some other posts, it looks like MTOM (not familiar with it) may be the solution, but I'm not sure how to configure it in my web methods or make it work.

Answer 1

You can try a few things, you will not specify which protocols you use or how your hosting, so I assume that it could be IIS7 and your use of soap. In the web service’s web.config file, you can add the following, which will increase the file size allowed for transmission without 404 error:

  <system.web> <httpRuntime executionTimeout="999999" maxRequestLength="2097151" /> ... </system.web> 

The second thing you might want to do in the web.config file of a web service is to allow large file transfers:

  <system.webServer> <security> <requestFiltering> <requestLimits maxAllowedContentLength="2000000000" /> </requestFiltering> </security> </system.webServer> 

Another possibility:

 <location path="Copy.asmx"> <!-- Name of you asmx --> <system.webServer> <security> <requestFiltering> <requestLimits maxAllowedContentLength="104857600"/> <!-- 100 megs --> </requestFiltering> </security> </system.webServer> </location> 

The main problem with sending byte [] over web services is that they fall into the SOAP body, which receives the encoding as the base line 64. Such encoding files increase the file size by two-thirds in the body of the soap (that is, a file of size 6 MB becomes a 9 MB file over the wire).

Another possibility is to “split up” into dividing your data into smaller segments before transferring, which may be all you need. ChunkSize (installed at 500KB) can be a key factor in improving download performance based on factors such as network speed, server resources, etc.

 /// <summary> /// Chunk the file and upload /// </summary> /// <param name="filename"></param> private void UploadVideo(string filename) { #region Vars const int chunkSize = 512000; byte[] bytes = null; int startIndex, endIndex, length, totalChunks; WS.UploadRequest objRequest = new WS.UploadRequest(); #endregion try { if (File.Exists(filename)) { using (WS.UploadService objService = new WS.UploadService()) { using (FileStream fs = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.Read)) { //// Calculate total chunks to be sent to service totalChunks = (int)Math.Ceiling((double)fs.Length / chunkSize); //// Set up Upload request object objRequest.FileName = filename; objRequest.FileSize = fs.Length; for (int i = 0; i < totalChunks; i++) { startIndex = i * chunkSize; endIndex = (int)(startIndex + chunkSize > fs.Length ? fs.Length : startIndex + chunkSize); length = endIndex - startIndex; bytes = new byte[length]; //// Read bytes from file, and send upload request fs.Read(bytes, 0, bytes.Length); objRequest.FileBytes = bytes; objService.UploadVideo(objRequest); } } } } } catch (Exception ex) { MessageBox.Show(string.Format("Error- {0}", ex.Message)); } 

Answer 2

ASMX Web Services is an outdated technology that should not be used for new development.

One of the missing features is support for streaming and large files. In particular, the message will be duplicated in memory up to four times on the way to the client.

WCF supports true streaming.

Answer 3

As @John Saunders explained, web services have poor support for streaming and processing large files, you can use compression to transfer the file through the web service instead of sending the raw file.