NHibernate Search - Multiple Web Servers

Question

I use the NHibernate.Search assembly and I am looking for best practices using this with multiple web servers.

We have enough space on our web servers to process the indexes that we generate, so I think the best route is to have indexes on every web server. Then the classes I'm indexing add the version column. My only question, if I do this, is NHibernate.Search smart enough to pull out the last entry and index it if we say that web server A updated the record and the web server index B is out of date?

Another option is to store indexes in a common file location and retrieve from this network resource. This is similar to a solution that is different from ideal, because it does not allow for greater redundancy.

How do others solve this problem with the NHibernate.Search and / or Lucene.NET indexes?

Answer 1

At the very moment when you decided to place your indexes on different machines, you presented the problem of "distributed search". Things like replication, redundancy, management, monitoring, search aggregation become important and interesting problems that need to be addressed.

However, Solr is one of the recommended solutions to the problem. Moreover, SolrNet can help you integrate it with Nhibernate.

I used both projects in conjunction with Nhibernate, and this may be a bit confusing at the beginning, but it will pay off later.

In your case, you can run Solr on your web servers.

Answer 2

We use the Master / Slave approach, which is supported by NHibernate Search and Lucene.net.

Each WebServer has a subordinate copy of the index and does not index.

Each time the web server updates something, it sends a message to the server service (we use Rhino ServiceBus with msmq), which performs indexing (by downloading the updated object and re-indexing it).

Every 10 seconds (we need modern search queries - the usual practice has a 30-minute grace period), the web server checks for new versions of the index and retrieves it if necessary. It works very well, since the changes are incremental, so the attraction of the full index is only required if we are doing optimization or general re-indexing.

If you need a higher speed, you can optimize it using the ram implementation on web servers, but with rather complicated wildcard searches at the 32 MB index, we are still below 10 ms for queries.

Another optimization would be for the web server to perform indexing, but only send an incremental copy to the backend to add to the main index. This would save the database call from the backend service, albeit with a certain degree of complexity, since you need to delve into the bowels of NHibernate Search / Lucene to do this.

Answer 3

Your first idea will not work. farm nodes do not interact with each other. they depend on shared resources like a second level cache. In addition, you do not need separate indexes because they will be desynchronized.

The general location of the files is the path. As for redundancy, I would think that storage redundancy is relevantly trivial, you can start with a RAID array