Background processes for custom tasks?

Question

Users on the website I create can request the availability of usernames on various social networks by entering a field and pressing return (see this website for an example ). When a user submits a name for verification, I must request availability from many other third-party services at the same time. Each availability check requires an HTTP request. This means that a single request from a user can trigger many HTTP requests on the server.

Now I would like to return the results to the user as quickly as possible. Thus, I want to perform each of these backend availability checks separately and return the results as quickly as I get them from third-party developers. I would also like to use background workflows to support downloading all of these HTTP requests from my server.

Is this a viable use of background workers, or should they be used only when the user does not expect results immediately (for example, sending emails)?

Is this the best way to architect this app?

Answer 1

For me, this looks like an ideal use case for a combination of WebSockets and a reactor framework , such as EventMachine or node.js.

Why WebSockets?

For the query part, this does not really matter. However, various external services will respond to various delays, which means that in order to present the results to the user as soon as possible, you will probably have to run a long polling request for each of these services (which usually blocks the processes that process these requests ) or use a sequence of queries with a long survey to receive answers as they become available. Each HTTP request has a certain amount of overhead required to establish a connection, and you will transmit much more data in the HTTP headers than in the response itself.

On the other hand, the WebSocket connection is established once and from that moment acts as a kind of bidirectional channel that can be used to send messages. These messages may be the responses of various services that will be transmitted to the client as soon as they arrive. This saves a lot of overhead and quickly responds to user requests.

Why a reactor framework?

If you use a background job to handle responses, you will most likely have fewer workflows than the number of requests. This means that some of the requests will have to wait until the workers are ready, so the user will receive a response later than he could receive if all the requests were executed in parallel. Asynchronous I / O allows you to simultaneously transmit all these requests and return the results to the user when they enter.

If you used a background job queue, you will also need to save the results to some kind of data store so your web server can poll it to find out when a particular request is completed. This server-side polling also increases the delay with which the user receives data.

Summing up: using the reactor structure + WebSockets will not only improve the user experience, but will also be easier to implement. Check out the socket.io library for node.js: it should allow you to implement your use case in a dozen lines of code or so.