Rails: Can I run background jobs on another server?

Question

Is it possible to host an application on one server and run queue jobs on another server?

Possible examples:

• Two different EC2 instances, one with a primary server and one with a queue service.

• Accept an application in Heroku and use an EC2 instance with a queue service

Is it possible?

thanks

Answer 1

Yes, definitely. We created delayed_job this way where I work.

There are several requirements for the job:

• Servers must have a synchronized clock. This is usually not a problem if the server time zones are configured the same.
• All servers must have access to the same database.

To do this, you simply use the same application on both (or all, if more than two) servers and run workers on any server that you want to process tasks. Any server can still set up queues, but only one (s) with working workers will actually process them.

For example, we have one interface server, a db server and several worker servers. The interface server serves the application through Apache / Passenger, connecting the Rails application to the db server. workers have the same application, although Apache does not work, and you cannot access the application through http. On the other hand, they have delayed_jobs workers. In a general scenario, an interface server stops jobs in db , and worker servers process them.

One caveat: if you rely on physical files in your application (attachments, log files, downloaded XML, or something else), you will likely need a solution like S3 to save these files. The reason for this is that actual files may not be available on individual servers. An example of this is that your user had to upload his profile picture to his web server, files are likely to be stored on this server. If you have another server for resizing profile images, the image will not exist on the production server.

Answer 2

Just propose another viable option: you can use a new working service like IronWorker , which relies entirely on a resilient cloud server farm inside EC2.

Thus, you can run / schedule tasks, and they will parallelize many threads spanning multiple servers - all without worrying about infrastructure.

The same deal with the database - it must be accessible from the outside.

Full disclosure: I helped build IW .