Distributed Video Encoding - Gearman vs Beanstalkd

Question

I would like to create a distributed video encoding unit for several dozen machines. I had never worked with a messaging queue before, but the 2 that I started playing with were Gearman and Beanstalkd.

Beanstalk seems a lot simpler and easier to use than Gearman, but it is not as much as a function.

One thing I don’t understand is ... how do you create new workers on all servers? I plan to use php. Is it as simple as working with work.php in the CLI with "&" and just sitting there waiting for work?

I noticed that the mechanism does not kill the process after completing the task, but Beanstalk does this, so I have to restart the script after every work on each server.

Currently, I am more likely to use Beanstalk, the general flow of things that I planned was:

Run cron on each server that checks for a predetermined number of workers. If this is less than expected, create new workflows. Each process will take approximately 2-30 minutes.

Maybe I have a flaw in my logic here? Let me know what would be the “best” or “right” way to do this?

Answer 1

Terminology I will only use to try to be clear ... There is a concept of producer and consumer. The producer creates jobs that are queued (i.e. beanstalk service), which is then read by the consumer.

There are several ways to record a consumer. You can either start the task every x period of time through the cron task, or simply start the consumer after 1 cycle through php (or whatever you have).

Where to install the service really depends on what you are going to do. For me, I usually install the service both on the consumer (s) and on a separate box (sometimes this is unnecessary depending on your needs).

If you want longevity on the queue side, you should use the Beanstalk (-b) binning option. If something happens to your beanstalk service, this will allow you to reboot with minimal data loss in the queues (if there is no information). Longevity on the manufacturer's side may be due to the fact that you have several queues.