I am trying to set up scalable background image processing using beanstalk. My setup is this:
- The application server (runs on Elastic Beanstalk) receives the file, places it on S3 and sends a request for its processing through SQS.
- The Worker server (also running on Elastic Beanstalk) checks the SQS queue, takes a request, downloads the original image from S3, processes it in 10 different variants and saves them back to S3.
- These loading events occur at a speed of about 1-2 lots per day, 20-40 snapshots of each batch at an unpredictable time.
Problem: I am currently using a single micro-instance for a worker. Generating one version of the image can take from 3 seconds to 25-30 (it seems that the first ones are executed in 3, but then the micro-instance slows down, I think this is its 2-second surge in workload). Anyway, when I upload 30 photos, which means that the task: 30 photos * 10 options each * 30 seconds = 2.5 hours to process ??!?!
Obviously, this is unacceptable, I tried to use a "small" instance for this, the performance there is consistent, but it is about 5 seconds per option, so still 30 * 10 * 5 = 26 minutes per batch. Still not very acceptable.
What is the best way to attack this problem, which will get the fastest results and be effective at the same time?
Solutions I can think of:
beanstalk. , , . . 1 , 1 70%, 30% - 1 . , , , , , , , . , (, ?), , beanstalk, , , .
beanstalk , SQS, , ( ), , , , . , . , beanstalk git, ..
-
, ?