SaltStack Monitoring

Question

Is there anything to monitor SaltStack installations other than halite? I installed it, but this is not quite what we are looking for.

It would be nice if we had a web guiy or even a daily email address that showed the status of all minions. I am very comfortable writing scripts, but I don’t know what to do with the script.

Does anyone have any ideas?

Answer 1

In the case of monitoring, you mean the current salt , you can try one of the following:

• SaltStack Enterprise GUI
• Foreman
• Saltpad
• Molten
• Halite (DEPRECATED by SaltStack)

This graphical interface lets you more than just know if minions are alive. They will allow you to work with them just as you would with a solo client.

And in the case of monitoring, you only mean if the salt master and salt minions work , you can use universal monitoring tools, such as:

• Icinga
• Naemon
• Nagios
• Shinken
• Sensu

In fact, these tools can track various services on hosts that they know about. the host can be any machine with an ip address, and the service can be any resource that can be requested through the base OS. An example of a host can be a server, a router, a printer ... And an example of a service can be a memory, disk, process, ...

Answer 2

Not an absolute answer, but we are developing a saddle that replaces and improves halite. One of its features is to show the status of all your minions. You can try: Saltpad Project Page on Github

Answer 3

You can take a peek at consul , while it’s not special to SaltStack, I use it to monitor that the salt master and the saltminions work on hosts that should be.

Another simple test would be to run something like:

salt --output=json '*' test.ping 

And compare between different runs. This is not an amazing monitoring, but at least it shows that your minions get up and communicate with your master.

Answer 4

Another option would be to use the salt.runners.manage functions, which comes with status .

To print the status of all known salt minions, you can run this on your salt host:

 salt-run manage.status salt-run manage.status tgt="webservers" expr_form="nodegroup" 

Answer 5

I had to write my own. As far as I know, there is nothing that will do this, and halite does not work for what I need.

If you know Python, it's pretty easy to write a salt monitoring application. For example, my application had a thread that from time to time updated the list of hosts from salt keys and several threads that ran various commands against this list to verify that they were. Monitor threads updated the timestamped dictionary with success / failure for each host after they started. It was hacked along with the color of the HTML code to display the status of each node. It took me around noon to write it.

If you don't want to use Python, you can painfully do something similar to this inefficient, fast, unchecked hack using the command line tools in bash.

 minion_list=$(salt-key --out=txt|grep '^minions_pre:.*'|tr ',' ' ') # You' for minion in ${minion_list}; do salt "${minion}" test.ping if [ $? -ne 0 ]; then echo "${minion} is down." fi done 

It would be easy enough to change the file for recording or send a warning.

Answer 6

halite depreciated in favor of the paid version of ui, sad but true - still the salt shell does the job. I just guess that your best monitoring will be the one you can write yourself, fortunately, there is a salt-api project (which I believe was part of halite, not sure about this), I would recommend that you use it with a tornado, since it is better than the cherry version .

So, if you need a good interface, you may want to work with api after setting it up ... when setting up a tornado, make sure that you are ok with authentication (I had some problems here), here is how you can check this:

Using Postman / Curl / whatever:

check if api is alive: - there are no messages (just look if api is alive) - get the request http: // masterip: 8000 /

login (you will need to take the token returned here to perform most operations): - send to http: // masterip: 8000 / login - (x-www-form-urlencoded data in the postman), raw:
Username: YOURUSERNAME
Password: YOUR PASSWORD
eauth: pam

• im using pam, so I have a user with your username and your passport added on my main server (as a regular user who works with pam)

get minions, http: // masterip: 8000 / minions (you will need to send a token from the login operation)

get all jobs, http: // masterip: 8000 / jobs (you will need to send a token from the login operation),

So basically, if you want to do anything with salt glass control, just play with this salt-api and get what you want, there are output formats in the salt box so you can get all the data even as json (if your interface is javascript) it allows you to run cmd or everything you need, and the monitoring is yours (if you don’t switch from the community to the pro version) or if you don’t want to use the specified salt bar (which, sorry, guys, is the last once updated a year ago in accordance with the repo).

by the way. you may need to change this port 8000 to something else depending on the version of the salt sink / tornado / config.