All geek questions in one place

Continuous Job Queue Delay for Java

I am looking for an existing system to replace the existing slow and complex independent written task management mechanism. Existing system:

  • 1 MySQL DB with a long massive job table - queue
  • Multiple servers (written in java), all retrieving jobs from the queue and processing them
  • the job was NOT removed from the queue after it was processed in order to restart it later
  • the task can create other tasks and insert them into the queue

Limitations:

  • As more and more jobs are created and inserted into the queue, it takes more time to retrieve jobs from it. (Tasks are selected by priority and type) - create a bottle neck

I am looking for an existing system that can replace this one and improve its performance. Any suggestions?

thank

+3
source share
2 answers

I do not recommend JMS at all , but it seems like this is really what you need here. Distributed, transactional, continuous job queue management is what JMS is.

Popular open source versions include HornetQ and ActiveMQ .

+2
source

You can:

  • submit your jobs to Amazon Simple Queue Service (maybe JAXB is sorting)
  • dynamically run multiple instances of EC2 according to the queue length and possibly
  • ( S3) ( JAXB ).

, , EC2 Spot, . , ;)

+2

Source: https://habr.com/ru/post/1755934/

More articles:


All Articles