All geek questions in one place

Log Viewer Database Selection

In the near future I will use the log viewer utility. But I stuck to choosing a database. My requirements are as follows:

  • 5 GB of data daily
  • 5 TB total data size
  • Search log data in less than 10 seconds

I know that PostgreSQL will work if I fragment tables. But can I get this work written above. As far as I understand, NoSQL is the best choice for storing logs, since logs are not very structured. I saw an example as shown below and using hadoop-hbase-lucene seems to be promising: http://blog.mgm-tp.com/2010/03/hadoop-log-management-part1/

But before deciding, I wanted to ask if anyone had made such a choice before and could give me an idea. Which DBMS is best suited for this task?

+4
source share
2 answers

My magazines are very structured :)

I would say that you do not need a database in which you need a search engine:

  • Solré is Lucene-based, and it packs everything you need together.
  • ElasticSearch is another Lucene search engine
  • Sphinx is nice that you can use multiple sources for each search index - enrich your raw logs with other events.
  • Facebook scribe for searching and collecting magazines

Update for @JustBob: Most of the mentioned solutions can work with flat files without affecting performance. They all need an inverted index, which is the hardest part to create or maintain. You can update the index in batch mode or online. The index can be saved in RDBMS, NoSQL format or in the usual flat file storage format (configured using a search engine)

+5
source

Here you can find a lot of information:

http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis

See what suits your needs.

In any case, NoSQL is the right choice for such a task.

You should also consider the MongoDB / CouchDB learning curve, although they do not perform functions such as Cassandra or Hadoop, they are easier to learn.

MongoDB is used by Craigslist to store old archives: http://www.10gen.com/presentations/mongodb-craigslist-one-year-later

+4
source

Source: https://habr.com/ru/post/1446891/

More articles:


All Articles