Rails background image loading causing application timeout

Question

Unfortunately, the award was given to an answer that does not solve this problem, for those who have similar problems.

I have an image upload form (heroku to s3). When I submit the form, my rails server waits for a background job that loads the image until it completes before returning a response to the user. This causes an application timeout every time there is an image download.

Current order of events:

• User Submits Form
• The server receives the form
• If there is an image, the server starts a background job
• If the background task has begun, the server waits for its completion (the time of the rails is here)
• If started, the background job ends
• The server is processing the request.
• The server is responding to the user

Desired order of events:

• User Submits Form
• The server receives the form
• Server processes fields without image
• If there is an image, the server starts a background job
• The server is responding to the user
• The background job runs and the server processes the downloaded image (saves the URL)

Bootloader code

class PhotoUploader < CarrierWave::Uploader::Base include ::CarrierWave::Backgrounder::Delay include CarrierWave::MimeTypes process :set_content_type storage :fog end 

Carrierwave :: initializer backgrounder

 CarrierWave::Backgrounder.configure do |c| c.backend :sidekiq, queue: :carrierwave end 

User model

 class User < ActiveRecord::Base mount_uploader :photo, PhotoUploader, delayed: true process_in_background :photo end 

The controller code is missing because the form is being processed by ActiveAdmin. I can redefine where necessary, but could not figure out what needs to be changed.

What do I need to change to get the correct order of events?

Answer 1

The main problem here is that Heroku has strict limits on how long the request can block without sending data back to the client. If you press this limit (30 seconds for the start byte), Heroku will delay your request. For file uploads, it is very likely that you will reach this limit.

The best approach is for the user's browser to directly upload the file to S3. There are several discussions that are relevant to this: Direct download to S3 using Carrierwave

If you use something like a jQuery file upload plugin ( https://github.com/blueimp/jQuery-File-Upload ), the stream will look something like this:

• The user adds one or more files to the form before clicking submit.
• Files are uploaded directly to S3, and a file upload token is added for each file to your form.
• The user submits a form with tokens of uploaded files, and not with the contents of the files.
• The server can move files to their real home on S3 based on the supplied token.

This allows your web server to focus only on serving the request, rather than downloading files, which can be time consuming.

This requires a bit more work due to Heroku limitations, but in the end I consider your only option to avoid timeout restrictions.

In addition, I would recommend that you create an S3 boot box and then set the S3 lifecycle policy to clear files that are out of date. When you do direct file downloads, usually some downloads are not processed due to user failure, etc. Therefore, the life cycle does the work of cleaning these files.