Running a command on a remote server with isolation, redundancy, and asynchronous

Question

I have several servers that require commands to be executed on other servers. For example, a Bitbucket sender bit server running git pull on another server. Another example is a CI server, which pulls out a new docker image and restarts the instance on another server.

I would usually use ssh for this, creating a user / group specifically for a job with limited rights.

A few minus with ssh :

• A synchronous ssh call means that git push will have to wait for completion.
• If the host is not connected for any reason, the ssh command will fail.
• Access to sudoers keys, users, and permissions can become cumbersome.

Several possibilities:

• Find open source source code (I tried until no luck)
• Configure the REST API on each server that accepts calls with a specific authentication type, for example. POST https: // server / git / pull /? Apikey = a1b2c3
• Configure Python / Celery to perform tasks in a different queue for each host. This means that each server runs celery, which can execute commands, and possibly a service that accepts REST API calls, converting them to Celery tasks.

Is there a good solution to this problem?

Answer 1

Problem definition

• You want to be able to start a remote task without waiting for its completion.

This can be achieved in any number of ways, including SSH. You can execute a remote command without waiting for its completion, closing or redirecting all input / output streams, for example. eg:

 ssh user@host "/usr/bin/foobar </dev/null >/dev/null 2>&1" 

2. You want to be able to postpone the task if the host is currently unavailable.

This requires a queue / retry system. You will also need to decide whether the destination hosts will request messages (pull), or whether messages will be sent to the destination hosts from another location (push).

3. You want to simplify access control as much as possible.

It is impossible to completely avoid this problem. One solution would be to put most of the authentication logic on a centralized task server. This divides the problem into two parts: setting access rights on the task server and setting authentication between the task server and the target nodes.

Solution Examples

• Hosts try to run tasks over SSH using the method above for asynchrony. If the host is unavailable, the task is written to the local file. The Cron task periodically repeats sending unsuccessful jobs. Access control through SSH keys.

• Hosts add tasks by writing commands to files on the SFTP server. The Cron task on the target hosts periodically checks for new commands and executes them if found. Access control is controlled using SSH keys on the SFTP server.

• Hosts submit tasks to the REST API, which adds them to the queue. Desmond celery on each target host spends from the queue and performs tasks. Access is controlled primarily by credentials sent to the task sequence server.

• Hosts submit tasks to the API, which adds tasks to the queue. Custom task nodes pull the tasks out of the queue and send API requests to the target nodes. Authentication, controlled by the cryptographic signature of the sender added to the request, is verified by the task server on the target node.

You can also see tools that perform some or all of the necessary functions out of the box. For example, some Google search queries came up with Rundeck , which seemed to have some job scheduling and REST API capabilities. You should also consider using any existing automated deployment or management tools already present on your system.

conclusions

Ultimately, there is no single correct answer to this question. It depends on your specific needs. Ask yourself: how much time and effort do you want to spend on creating this system? How about the service? How reliable should it be? How much do you need to scale? And so on, to infinity ...