Sitecore Lucene indexing - file exception not found using extended database crawler

Question

I have a problem with Sitecore / Lucene in our content management environment, we have two content delivery environments where this is not a problem. I use Advanced Database Crawler to index multiple elements of specific patterns. A pointer points to the main database.

The index will remain “stable” for several hours or so, and then in the logs I will start to see this error. However, if I try to open Searcher.

ManagedPoolThread #17 16:18:47 ERROR Could not update index entry. Action: 'Saved', Item: '{9D5C2EAC-AAA0-43E1-9F8D-885B16451D1A}' Exception: System.IO.FileNotFoundException Message: Could not find file 'C:\website\www\data\indexes\__customSearch\_f7.cfs'. Source: Lucene.Net at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run() at Sitecore.Search.Index.CreateReader() at Sitecore.Search.Index.CreateSearcher(Boolean close) at Sitecore.Search.IndexSearchContext.Initialize(ILuceneIndex index, Boolean close) at Sitecore.Search.IndexDeleteContext..ctor(ILuceneIndex index) at Sitecore.Search.Crawlers.DatabaseCrawler.DeleteItem(Item item) at Sitecore.Search.Crawlers.DatabaseCrawler.UpdateItem(Item item) at System.EventHandler.Invoke(Object sender, EventArgs e) at Sitecore.Data.Managers.IndexingProvider.UpdateItem(HistoryEntry entry, Database database) at Sitecore.Data.Managers.IndexingProvider.UpdateIndex(HistoryEntry entry, Database database) 

From what I'm reading, this could be related to updating the index while there is an open reader, and when the merge operation occurs, the reader will still have a link to the deleted segment or something like that (I 'm not an expert on Lucene).

I have tried several things without success. Including a subclass that separates the Sitecore.Search.Index object and overrides CreateWriter (bool recreate) to change the scheduler / merge policy and adjust the merge factor. See below.

 protected override IndexWriter CreateWriter(bool recreate) { IndexWriter writer = base.CreateWriter(recreate); LogByteSizeMergePolicy policy = new LogByteSizeMergePolicy(); policy.SetMergeFactor(20); policy.SetMaxMergeMB(10); writer.SetMergePolicy(policy); writer.SetMergeScheduler(new SerialMergeScheduler()); return writer; } 

When I read the index, I call SearchManager.GetIndex (Index) .CreateSearchContext (). Finder, and when I finished getting the documents I needed, I call .Close (), which I thought would be sufficient.

I thought I might try redefining CreateSearcher (bool close) to open a new reader every time, and I will leave after that. I really don't know enough about how Sitecore handles Lucene, its readers / writers?

I also tried to play with the UpdateInterval value in the web config to find out if this would help, alas, this did not happen.

I would be very grateful to everyone who: a) knows about any situations in which this can happen, and b) any potential tips / solutions, as I start to bang my head against a rather large wall :)

We are launching Sitecore 6.5 rev111123 with Lucene 2.3.

Thanks,

James.

Answer 1

It seems that Lucene is fading away when you try to reindex what is already in the indexing process. To verify this, try the following:

• Set the updateinterval your index to a really high value (8 hours).

• Then stop w3wp.exe and delete the index.

• After deleting the index, try rebuilding the index in Sitecore and wait for it to complete.
• Repeat the test and check if this has happened.

If this does not happen anymore, it will be too low updateinterval , which will cause your index (which is probably still being created) to be overwritten with a new one (this also will not be completed) by calling segments.gen to contain incorrect index information .

This .gen file will .gen your indexreader which segments are part of your index and recreated after the index is restored.

This is why I suggest trying to disable updates for a large amount of time and rebuild it manually.