Warning / notification using Kibana3?

Question

I use logstash-1.4.1 , elasticsearch-1.1.1 and kibana-3.1.0 to analyze my logs. I can view and request logs.

Where warning / notification is required when a specific log / event occurs. For example: when the logon logon failure repeats again and again, a notification / notification is required (pop-up, mail, etc.).

Currently, I can request my log, for example, for Failed login, but I want that whenever such a log appears, a notification / pop-up window appears, than I manually request it.

Can this be done using the above three? How can this be achieved?

Answer 1

In logstash there is an email option in which when a specific template is found in the log, an email can be sent. Check out the docs for further reading: http://logstash.net/docs/1.4.1/outputs/email

Answer 2

You can use Observer to monitor your Elasticsearch. He notifies you by mail.

For more information see this link:
https://www.elastic.co/products/watcher

You can follow these steps to configure Watcher:

Step 1. Install the plugin for Watcher (for version 1.7).

 bin/plugin --install elasticsearch/watcher/latest bin/plugin --install elasticsearch/license/latest 

Step 2. Restart Elasticsearch

 ES_HOME/bin/elasticsearch 

Step 3. To verify that Watcher is configured.

 curl -XGET 'http://localhost:9200/_watcher/stats?pretty' 

Step 4. CODES For log data for errors.

  PUT /_watcher/watch/log_error_watch { "trigger": { "schedule": { "interval": "10m" } }, "input": { "search": { "request": { "indices": [ "logs" ], "body": { "query": { "match": { "message": "error" } } } } } }, "condition": { "compare": { "ctx.payload.hits.total": { "gt": 0 } } }, "actions": { "send_email": { "email": { "to": "<username>@<domainname>", "subject": "Cluster logs", "body": "Cluster Error Logs ", "attach_data": true } } } } 

Step 5: to configure email, enter the lines below in elasticsearch.yml:

  watcher.actions.email.service.account: work: profile: gmail email_defaults: from: <email> smtp: auth: true starttls.enable: true host: smtp.gmail.com port: 587 user: <username> password: <password> 

Step 6. REMOVING A TEACHER

 curl -XDELETE'http://localhost:9200/_watcher/watch/log_error_watch' 

Answer 3

You can control elasticsearch with "Watcher" (one of their products).

Here is a link to product information: https://www.elastic.co/products/watcher

And on the next page you can check the product manual: https://www.elastic.co/guide/en/watcher/current/index.html

Answer 4

There is a looooong thread discussing this as a complement to Kibana (although they naturally focus on Kibana 4 at the moment).

Current status: No, not yet, and not yet planned. But there are some options that are mentioned:

https://github.com/Yelp/elastalert

and

https://bosun.org/expressions.html#logstash-query-functions

Answer 5

Here's how to perform email alerts and monitoring with updated ES and Kibana. I am using elasticsearch-5.5.0 , kibana-5.5.0 using XPack and Watcher.

Step 1. Install XPack for Elasticsearch and Kibana

 bin/elasticsearch-plugin install x-pack bin/kibana-plugin install x-pack 

Step 2. Restart ES and Kibana

 ./bin/elasticsearch ./bin/kibana 

Step 3. Set up your email account in elasticsearch.yml

 xpack.notification.email.account: outlook_account: profile: outlook email_defaults: from: <sender-email> smtp: auth: true starttls.enable: true host: smtp-mail.outlook.com port: 587 user: <username> password: <password> 

** I tried it with a candle, and everything worked out fine. Just changed the profile to sparkpostmail and the host to smtp.sparkpostmail.com. You can find a guide for other email settings: https://www.elastic.co/guide/en/x-pack/5.6/actions-email.html#configuring-email-actions

Step 4: Configure email actions in Kibana Dev tools (you can do this as a curl command)

 PUT _xpack/watcher/watch/error_report { "trigger": { "schedule": { "interval": "1h" <OR TIME INTERVAL TO MONITOR AND ALERT> } }, "input": { "search": { "request": { "indices": [ "logs" ], "body": { "query": { "match": { "message": "error" } } } } } }, "actions": { "send_email": { "email": { "to": "<YOUR EMAIL>", "subject": "Cluster logs", "body": "Cluster Error Logs ", "attach_data": true } } } } 

OR! If you want to configure Kibana to send the panel or visualization via email, configure the following email action:

 PUT _xpack/watcher/watch/error_report { "trigger" : { "schedule": { "interval": "<TIME_INTERVAL>" } }, "actions" : { "send_email" : { "email": { "to": "<YOUR EMAIL>", "subject": "Error Monitoring Dashboard", "attachments" : { "error_dashboard.pdf" : { "reporting" : { "url": "http://<YOUR_HOST>:5601/api/reporting/generate/dashboard/<DASHBOARD_ID>?_g=(time:(from:now-7d%2Fd,mode:quick,to:now))", // This is where you configure settings like time interval "retries":6, "interval":"15s", "auth":{ "basic":{ "username":"<USERNAME>", "password":"<PASSWORD>" } } } } } } } } } 

Step 5 (optional). Remove the observer when done with Kibana Dev Tools.

 DELETE _xpack/watcher/watch/log_error_watch 

This is just a brief update of the above answer for kibana and xpack updates, so all in one place! Thanks