Setting up Elasticsearch server for processing data from microservices

Question

I am very new to finding elastics and scaling it, and I have a question that I don’t even know how to approach.

Here's the situation:

There are several servers with Rails microservice applications. Each of them receives each its own rather large amount of data (more specifically, aggregating messages from different social networks, so indexed search fields are the same in all databases).

I need to find a solution that allows me to store data where it is, and set up an elasticsearch server designed exclusively for searching through multiple databases, without the corresponding Rails applications included on this search server. This potentially means setting up ES on each of the other servers, defining search patterns there, but performing a search on multiple patterns on a completely different server.

The ultimate goal of these manipulations should be to send all ActiveRecord objects / or all associated attributes to the main application.

Is it even possible to achieve? Maybe someone had a similar problem?

I lost a little about how to start with it.

Answer 1

This question is a bit broad, but I think I can at least point you in the right direction from what I understand. First, let me start by defining your problem as I understand it.

You have several databases populated with their own microservice. Each database contains similar information that you want to find (for example, author, body, title, etc.). You want an elasticsearch cluster that has access to data in all of these databases to be able to return a result that includes the correct databases and the document that matches the search.

Elasticsearch is very effective when it comes to complex cases like this. Since all your data has a similar structure and fields, you can simply use one index with additional fields to store it, from which the document is created, and the document identifier from this database. This will allow you to perform searches such as "Give me all the notes made by William Shatner through these 3 social networks."

You will need a few extra features to complete this work. First, you need a mechanism to get data from the database into the search index. In my team, we use a separate IndexingService, which knows how to read event streams and send current data to the ES index. You just need to define an indexing strategy (for example, how often do you update an index with new records?). Secondly, you will need client-side logic to take the initial search result and get the corresponding record from the database.

This is just one way to solve your problem. If you need an approach that allows you to maintain a different index for each social network, but still has a central place, you can search for everything that I propose to use in the use of Elasticsearch Tribe nodes. Basically, this is the only place to send a search that knows about each search cluster and how to interact with them to return a single search result.

The best way to learn elasticsearch is to simply start the cluster and start experimenting! Good luck