Lucen from memory

Question

I am using Lucene v4.10.4. I have a rather large index, it can be more than a few GB. Therefore, I get OutOfMemoryErrorwhen initializing IndexSearcher:

try (Directory dir = FSDirectory.open(new File(indexPath))) { 

    //Out of Memory here!
    IndexSearcher searcher = new IndexSearcher(DirectoryReader.open(indexDir));

How do I tell Lucene DirectoryReader not to load more than 256 MB at a time?

Magazine

Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.fst.BytesStore.<init>(BytesStore.java:68)
    at org.apache.lucene.util.fst.FST.<init>(FST.java:386)
    at org.apache.lucene.util.fst.FST.<init>(FST.java:321)
    at org.apache.lucene.codecs.blocktree.FieldReader.<init>(FieldReader.java:85)
    at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:192)
    at org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:441)
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:197)
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:254)
    at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:120)
    at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:108)
    at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:923)
    at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:53)
    at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:67)

Answer 1

You must first check the current heap size of your JVM.

java -XX:+PrintFlagsFinal -version | grep MaxHeapSize

If this number is not suitable for your use case, you should increase it when starting your program using the -Xmx option for the java command. An example command for assigning 8 GB of heap memory would look like this:

java -Xmx8g -jar your_jar_file 

Hope this helps.