How can I upload images in bulk (about 100-200 images at a time) in Ruby on Rails?

Question

I want to upload images (about 200 kB each) in bulk. We have several options, such as CarrierWave, Paperclip and others. How can I do these downloads in bulk?

Answer 1

TL DR

not to do. Rails is not optimized for bulk upload, so do it out of range when you can.

Use FTP / SFTP

The best way to handle large file sizes is to use a completely out-of-band process rather than linking your Rails process. For example, use FTP, FTPS, SCP, or SFTP to upload your files in bulk. Once the files are on the server, process them using the cron job or use inotify to run the rake command.

Note. Be sure to pay attention to file locking when using this technique.

Use queue

If you insist on doing this through Rails, don't download hundreds of files. Instead, upload one archive containing your files that will be processed by the background job or queue. There are many alternatives, including Sidekiq and RabbitMQ among others.

After downloading the archive and sending the delivered queue, the queue process can unpack the archive and do everything that needs to be done. This type of solution scales very well.

Answer 2

Like other things in a Sc computer, the answer depends on that. I really mean

• Will end users download them ?. If so, use the jQuery File Upload plugin to present an easy to use interface.
• For storage, you can store it on your server. or even better, upload images directly from users' computers to amazon s3. here is an example Uploading files to S3 in Ruby using Paperclip
• Obviously, you will need to convert this to a background task, where all images will receive ajax images in separate tasks. if you don’t already have fav que, I would suggest resque or sidekiq

Note. If you choose to upload images directly to S3 via CORS , then your rails server will be free from file upload control. And direct downloads are recommended if you have large images or a large number of files to download.

However, direct downloads limit your ability to resize images (resize, etc.). so keep this in mind if you are choosing a direct download solution.