RoR - Loading large files on rails

Question

I have a rails webapp that allows users to upload videos where they are stored in the directory installed on NFS.

The current setup is suitable for small files, but I also need to support large file downloads (up to 4 GB). When I try to download a 4gb file, this ultimately happens, but it's terrible from a UX perspective: the download starts and the progress is displayed based on the XHR progress events, but then after 100% there is still a long wait (5+ minutes) before the server will respond to the request.

Initially, I thought that this was due to copying the file from some temp directory to the destination directory based on NFS. But now I'm not sure. After adding a log to my routes, I see that there is about a 3-minute wait between when the file download progress reaches 100% and when the code is executed in the action of my controller (before I do any processing to move the file to the NAS) .

I am interested in the following:

• What happens during this 3-minute wait after the download is complete and before my action is called?
• Is there a way for me to take into account what happens during this period so that the client receives a response immediately after the download is completed so that they do not exit the game?
• How do large files typically load in Rails? It seems like it would be a common problem, but I can not find anything on it.

(Note: I initially used CarrierWave to load when I discovered this problem. I deleted it and simply processed the save file using FileUtils directly in my model to make sure that the wait time was not the result of some CarrierWave magic happening behind the scenes, but got exactly same result.)

ruby -v: 1.9.3p362

rails -v: 3.2.11

Answer 1

I finally found the answer to my main question: What happens during this 3-minute wait after the download is complete and before my action is called?

All of this was explained very clearly in this post: Rails Path - Uploading Files

"When the browser downloads the file, it encodes the contents in the format" multipart mime "(the same format used when sending an email attachment). For your application to be able to do something with this file, rails must cancel this encoding. To do this "You need to read the huge request body and match each line with a few regular expressions. It can be incredibly slow and use a huge amount of processor and memory."

I tried the modporter Apache module mentioned in the post. The only problem is that the module and its corresponding plug-in were written 4 years ago, and it no longer works with their website, there is almost no documentation.

With modporter, I wanted to specify my NFS-based directory as PorterDir, in the hope that it would transfer file permissions along with the NAS without additional copying from a temporary directory. However, I could not go this far, because the module seemed to ignore my specified PorterDir and return a completely different path to my actions. In addition, the path that he was returning did not even exist, so I had no idea what was really happening with my additions.

My workaround

I had to quickly solve the problem, so at the moment I went with a somewhat hacky solution, which consisted of writing the appropriate JavaScript / Ruby code to process the downloaded files.

JS example:

var MAX_CHUNK_SIZE = 20000000; // in bytes window.FileUploader = function (opts) { var file = opts.file; var url = opts.url; var current_byte = 0; var success_callback = opts.success; var progress_callback = opts.progress; var percent_complete = 0; this.start = this.resume = function () { paused = false; upload(); }; this.pause = function () { paused = true; }; function upload() { var chunk = file.slice(current_byte, current_byte + MAX_CHUNK_SIZE); var fd = new FormData(); fd.append('chunk', chunk); fd.append('filename', file.name); fd.append('total_size', file.size); fd.append('start_byte', current_byte); $.ajax(url, { type: 'post', data: fd, success: function (data) { current_byte = data.next_byte; upload_id = data.upload_id; if (data.path) { success_callback(data.path); } else { percent_complete= Math.round(current_byte / file.size * 100); if (percent_complete> 100) percent_complete = 100; progress_callback(percent_complete); // update some UI element to provide feedback to user upload(); } } }); } }; 

(forgive any syntax errors just by typing it from my head)

On the server side, I created a new route for receiving file fragments. When sending the first fragment, I create the upload_id file based on the file name / size and determine if I already have a partial file from the interrupted download. If so, I will return the next start byte, which I need along with id. If not, I save the first piece and pass the identifier.

The process with the addition of additional fragments adds a partial file until the file size matches the size of the original file. At this point, the server responds temporarily to the file.

Then javascript removes the input file from the form and replaces it with hidden input, whose value is the path to the file returned from the server, and then submits the form.

Then finally, on the server side, I handle moving / renaming the file and saving its final path to my model.

Phew

Answer 2

You can use MiniProfiler to better understand where time is wasted.

Large file downloads need to be processed in the background. Any controllers or access to the database should simply note that the file has been downloaded, and then queue the background processing job to move it, and any other operations that may occur.

http://mattgrande.com/2009/08/11/delayedjob/

This article makes sense; each implementation will be different.