How to control a large hbase seal

Question

I want to keep an eye on the main hbase compaction. For each table, I would like to know

• at startup (via hbase or user)
• when he finished

How can I control this?

• through jmx?
• via user interface

In general, what additional parameters should be monitored to find that a basic seal is necessary?

Answer 1

HBase exports many metrics through Apache Hadoop scorecards. These metrics are also displayed through jmx. So ganglia or jmx are pretty simple ways to start looking at the metrics that HBase exports. Other options include OpenTSDB and propriety records.

Inside the ganglia or jmx, the metric you want to look at is hbase.regionserver.compactionQueueSize. This is the number of scheduled transactions.

The RegionServer user interface also includes a copy of some metrics, including compactionQueueSize. Later versions of HBase will look better, but this gives an example.

Answer 2

I know the answer is a bit late, but instead of manually viewing the log files, you can also try Hannibal. This is an open source tool that provides history charts for each region. The graphs show when compromises occurred and how long they lasted.

https://github.com/sentric/hannibal/

Answer 3

This applies to hbase versions 0.96 and 0.98

When the compaction started and the state of HBase Compactions can be gleaned from: <hbase.master.ip>:<hbase.master.info.port>/dump i.e. 10.4.4.40:60010/dump

in a section called

servers:

  <hostname>,60020,1406845954503: ...<other stuff here>... totalCompactingKVs=9121040186, currentCompactedKVs=14351361679, compactionProgressPct=1.5734347,... 

compactionProgressPct gives the percentage of completed compaction that you are currently deploying.

Here's a much friendlier way to view images:

http://<hbase.regionserver.ip>:<hbase.regionserver.info.port>/rs-status#queueStats

This will give you statistics that look like this:

Answer 4

Compaction status can be viewed from the Hbase log or the GUI Regionservers. But if we have huge transactions, it is difficult to track the state of the compaction area from the log. The GUI only shows compaction status on the same node and is tedious for viewing in a graphical interface. If we have "n" regionservers

Download a utility that collects the status of the online compaction region in Regionservers for this table. Github link in StatusUtil.java

Answer 5

We can get compaction status using JMX.

compactionQueueSize, compactionSizeNumOps, compactionSizeAvgTime, compactionSizeMinTime, compactionSizeMaxTime, compactionTimeNumOps, compactionTimeAvgTime, compactionTimeMinTime, compactionTimeMaxTime

Enable JMX metrics as shown at http://hbase.apache.org/metrics.html

You can download the JMX example here