Get closer to real time with chaos

Question

I need good links to use Hadoop for real-time systems, such as search with a short response time. I know that hadoop has overhead on hdfs, but what better way to do it with hadoop.

Answer 1

You need to provide much more information about the goals and objectives of your system in order to get good advice. Perhaps Hadoop is not what you need and you just need some distributed foo systems? (Oh, and you are absolutely sure that you need a distributed system? There you can do a lot with a replicated database on top of several machines with large memory.)

, .

• HBase, HDFS, Google BigTable. http://hadoop.apache.org/hbase/
• , . Gizzard, , : http://github.com/twitter/gizzard
• . , , , - Lucandra - Lucene Cassandra ? http://github.com/tjake/Lucandra

, - , , , - Thrift , , . , , , , . Hadoop, , , .

Answer 2

Hadoop - . , .

FWIW, HDFS . , Hadoop jar node, , , , .. ..

Answer 3

, . , , , , Lucene + SOLR . Hathi Trust , .

This is a completely different problem if the index changes in real time. Even Lucene will have problems updating the index, and you will have to search for search engines in real time. Some attempts to remake Lucene in real time and may need to work. You can also watch HSearch, a real-time distributed search engine built on Hadoop and HBase, hosted at http://bizosyshsearch.sourceforge.net