What is the best way to implement Email Alerts in Elastisearch?

Question

We will create a new job type site that will work in AWS, and we use Elastisearch for all job search and job search functions.

Email alerts will appear on the site. 1) Applicants can set an alert so that a new task that is posted according to certain keywords and is within X miles of a particular zip code will be sent by email. 2) Recruiters will be able to set alerts so that a summary with specific keywords is sent to them by email within X miles of a particular zip code.

Is there any open source code that will launch us?

I read a little about Watcher from Elastic.co, but we download and try to find an inexpensive solution. I will ping Elastic to feel their initial costs ...

Has anyone built a scalable web application that can handle 10,000 alerts per day using Watcher or some other Alerting tool?

thanks,

Brent Byers

Answer 1

If you don't want to use Watcher (yet), the next available option is to study the percolate API (+ an older blog post on this topic ) to implement the watch part and handle the alert part in your own code

Some examples of what people do there:

1. You might want to check out ElastAlert, created by Yelp and which should probably help you.

2. Toplog uses the percolate API to receive alerts when certain log templates are indexed. They also created a percolator Logstash in order to filter the document immediately after indexing it and save the corresponding percolated requests in redis (or whatever you have). Some people at Logstash adhere to the same idea and may find this plugin an acceptable option.

3. There is another example where an email is sent whenever a new product that matches a given request is indexed.

If you dig a little more, you will surely find many more examples. But the general idea is always the same:

1. Store Jobs / Candidate Requests Using the Percolate API
2. Submit new job / candidate documents
3. If matching requests are found in step 2, send alerts.

Finally, in Kiban there is an open problem for the integration of the warning system, which you might also want to see.

UPDATE (2016/05/04)

I decided to create a Logstash filter that supports the ES Percolate API. This is similar to what the guys from Toplog do, but it works as a filter to enrich the event, and not as a way out for redis.

You can find more information at: https://github.com/elastic/logstash/issues/2187#issuecomment-216760668