HBASE-ZOOKEEPER error: too many connections

Question

I use the Hbase-Hadoop combination for my application along with Data Nucleus as ORM.

When I try to access hbase through multiple threads at a time. It throws exceptions:

Exception in thread "Thread-26" javax.jdo.JDODataStoreException org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable javadoc for more information. Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase 

If necessary, I can provide a full stack trace (since a full stack trace makes it useless here).

Please help me with some conclusions on how to deal with this situation. Is there any configuration I need to do to increase the connection pool?

Answer 1

Zookeeper servers have a limit on active connections, the default is 30. You must increase this limit by setting the maxClientCnxns property, respectively, in the zookeeper configuration file, zoo.cfg.

For 100 connections:

 maxClientCnxns=100 

To inform zookeeper that the number of connections is unlimited:

 maxClientCnxns=0 

Answer 2

For the above problem, you need to restart the main HBase service. The team for her is

 sudo /etc/init.d/hadoop-hbase-master start 

this must be done before entering the HBase shell.

Answer 3

I do not suggest blindly dropping the maximum connection, this configuration is not the maximum connection to ZK, but the maximum number of simultaneous connections per client to ZK, according to the ZK document:

maxClientCnxns (no Java system property)

Limits the number of simultaneous connections (at the socket level) that one client identified by IP address can make to one member of the ZooKeeper ensemble. This is used to prevent certain classes of DoS attacks, including running out of file descriptors. Setting this value to 0 or deleting it completely eliminates the restriction on simultaneous connections.

therefore, you must have more than 30 concurrent connections to the ZK from the same IP address. in this case, use netstat to debug what this IP address is and verify that it can initiate these multiple connections.