Can PHP-FPM be used to manage queue consumers?

Question

There, the beanstalkd queue, which is filled with many tasks, says every 10 minutes, and the priority is that each task is processed as soon as possible. The task may take more than a few milliseconds, because there are calls to third-party services that from time to time have a timeout.

So, since PHP does not have multithreading, one option would be to create a large number of unoccupied workers who will try to reserve a task, but probably require too much RAM, which may not be available in these blocks,

Is it good to use PHP-FPM to configure the number of workers and save some RAM? Is it ready? Are there any better solutions?

thanks

Answer 1

I am launching a queuing system that deals with millions of messages per day. Mostly through Amazon SQS, but right now I'm also launching a new Beanstalkd system with over 600,000 messages.

As described on the topic blog , I have shell scripts running in loop processing messages (a loop in a PHP script to run multiple jobs before returning is also useful, at least for small jobs).

These shell scripts start with Supervisord . There is another blog entry about using this . Currently, I run more than 800 working scripts (for several different types of tasks) on nine machines, all are pulled from different queues and put the data back into other queues, writing to the database or files. The increase in the number of workers per machine is associated with an increase in the number of "numprocs" (or already large enough), and then, starting with more requests. You could also say 5 autostarts, and then another block of 50, which are ready to be launched as needed.

I find that each worker takes only about 20 mb of non-shared memory (the rest is common to processes). It depends on the tasks that, of course, the employees perform. Resizing images can be a lot of work. Partly because of this, I have a setting that allows you to frequently restart the PHP script.

Answer 2

Whenever I had to run stuff at the same time (or asynchronously), I sent gearman jobs to the workers. I usually had at least one process for each processor core on each physical computer.

PHP-FPM is a cgi daemon. So basically you will have a beanstalkd processor that runs many HTTP requests to your own system. You may need to go through your HTTP stack. Not sure if this is such a great idea.

You can also check pcntl_fork to deploy the current process to several running processes.