Scalable job queuing system for large-scale task scheduling

Scenario:

TL DR - I need a queuing system to run jobs based on a future timestamp, NOT in the order in which it is inserted.

I have a MySQL database that details the specific events that need to be performed (which will consist mainly of a series of arithmetic calculations and inserting / updating the database) in an exact sequence based on timestamps. The insertion time and when the event is completed has no correlation and is determined by external factors. The table also contains a second column of milliseconds, which increases the accuracy of synchronization.

This table is part of a job queue that will contain records that will run anywhere from a few seconds to several days in the future, and can potentially have up to a thousand records added every second. Should the queue be parsed continuously (every second?) - perhaps by selecting all the timestamps that have expired during that second, sorting by milliseconds, and then executing each event that is detailed using records.

Problem

Currently, the backend is completely written in PHP on an Apache server with MySQL (i.e. the standard LAMP architecture). Right now, the only way I can come up with to achieve what I have pointed out is to write a custom PHP script job queue that will parse and execute, each loop through this method . There are no other job systems that I know of that can set up queues according to a given timestamp / millisecond, rather than an entry time.

, , - MySQL- - , , script.

, , , , , , .

LAMP, - , , , ?

?

, PHP script ? "".

RabbitMQ , , , , "" - - , . , - , . , RabbitMQ, . ?

+4
1

, , . MySQL , . , RabbitMQ.

, , . "" , . , . , " " "" , "", , .

, :

  • , , .
  • , , .
  • , - , . .

, . RabbitMQ , . , Iron.IO StormMQ

0

Source: https://habr.com/ru/post/1534439/


All Articles