Can I Predict How Big My Zend Framework Index Will Be? (and some quick Q: s)

Question

I have around 100thousand rows in the mysql table, where each row has about 8 fields.

Finally, I figured out how to use Zend Lucene to index and search data from a mysql table.

Before I fully implement this functionality on my website, I have a few questions:

1- Is it possible to pre-determine the size of the index? This is because the Zend manual states that the maximum index size is 2 GB. I immediately think that this is not enough for my table!

2- I read posts where they say that finding Zend Lucene is very slow on large indexes, up to minutes! Is it faster to use mysql commands (SELECT, LIKE, etc.) instead of zend?

3. Are there any other solutions to my problem that should create a search engine for ads in which these functions at least do not require full-text mysql indexes (fields).

thanks

Answer 1

SOLR is basically an Apache Tomcat container that implements the REST interface for querying the Apache Lucene index. Yes, you should be able to run a Java application on your web server. This is a problem for you so that you can work with your hosting provider.

Customers using your web application do not need to run Java. Your PHP application can query the REST SOLR service and format the results in HTML format. The client sees only HTML output; he never needs to know that the data comes from a service implemented in Java.

Zend_Search_Lucene is a pure-PHP implementation that should work identically with Apache Lucene. Zend's solution even uses the same index file format. Therefore, they must be the same.

I used Java Lucene to index a StackOverflow data dump (October 2009). I indexed 1.5 million lines, including about 1 gig of text. The Lucene index was 1323 MB, while the MySQL FULLTEXT index with the same data was only 466 MB.

Using SQL LIKE predicates instead of any full-text indexing solution does not require any space, because in any case, it cannot use a regular index. But in my tests using LIKE it was about 200 times slower than Java Lucene, which in turn was 40% slower than the MySQL FULLTEXT index for the same data.

See my recent presentation on MySQL full-text indexing solutions:

http://www.slideshare.net/billkarwin/practical-full-text-search-with-my-sql

Not surprisingly, it cannot match the performance and scalability of Java Lucene technology. The advantage of PHP as a language improves development efficiency rather than operational efficiency.

---

update: I just tried to create an index using Zend_Search_Lucene . Creating an index is much slower with PHP than with Java Lucene technology, so I only indexed 10,000 documents. It took almost 15 minutes, which would take about 36 hours to index the entire collection. Compare this to Java Lucene, which in my test indexed a complete collection of 1.5 million documents in less than 7 minutes.

The size of the index created using Zend_Search_Lucene is 8.75 MB. By extrapolating this 150x, I believe that the full index will be 1312.5 MB. Therefore, I came to the conclusion that Zend_Search_Lucene creates an index about the same size as the index created by Java Lucene. This is as expected.