Election leader application with java client consul

Question

I'm trying to talk about how to conduct the election of application leaders using Consul. I am using LeaderElectionUtil from the java client consul.

I can choose a leader, and all nodes agree with the leader, but if the leader application dies, other nodes do not seem to know and continue to hold the dead leader when calling getLeaderInfoForService - that is, there are no new leadership choices.

Leader Leadership Guide ( https://www.consul.io/docs/guides/leader-election.html ):

"Note that the session only uses the gossip failure detector by default. That is, the session is considered to be saved by the node if, by default, the Surf health check did not declare the node unhealthy. If necessary, you can specify checks.

So, from this, I assume that perhaps I need to add an application level health check (TTL, etc.) to the session so that the session is invalid when the application terminates with an error? Is this the right idea, and if so, can this be done through the java client? Am I fine with the LeaderElectionUtil groove and writing my code to select a leader, but it seems that even in SessionClient there is no way to create a session with a health check associated with it?

Or maybe there is a better way to achieve this (detecting application-level failure to re-elect a leader)? I'm kinda stuck, so any pointers would be appreciated.

Answer 1

So I decided this in case anyone else comes across this problem.

I could not use LeaderElectionUtil, but I created my own class to do the same, but in the createSession method I added a TTL of 10.

private String createSession(String serviceName) { final Session session = ImmutableSession.builder().name(serviceName).ttl("10s").build(); return client.sessionClient().createSession(session).getId(); } 

For this to work, you will need a background thread that causes the session to be updated in the session at least once every 10 seconds.

Answer 2

I am trying to fulfill the same requirement: I have a Java service that should choose a leader, and I do not have the health settings configured in Consul.

Using LeaderElectionUtil in Consul-client is problematic, because if all the reasons mentioned above. Unfortunately, it is also impossible to configure LeaderElectionUtil , because all its internal actions are performed using private methods (it should have used protected and allow users to override session creation, for example).

I tried to implement "Service Registration" as described in "Basic Use - Example 1" in the README consul client, but calling AgentClient.pass() always changes the exception for me.

So, my solution - this is exactly what you indicated - do a session with TTL and update it until the service is up and running.

Here's my implementation, which requires the user to also register a callback, which is used to check if the service is valid for renewal, just in case:

 public class SessionHolder implements Runnable { private static final String TTL_TEMPLATE = "%ss"; private Consul client; private String id; private LinkedList<Supplier<Boolean>> liveChecks = new LinkedList<>(); private long ttl; private boolean shutdown = false; public SessionHolder(Consul client, String service, long ttl) { this.client = client; this.ttl = ttl; final Session session = ImmutableSession.builder() .name(service) .ttl(String.format(TTL_TEMPLATE, ttl)) .build(); id = client.sessionClient().createSession(session).getId(); Thread upkeep = new Thread(this); upkeep.setDaemon(true); upkeep.start(); } public String getId() { return id; } public void registerKeepAlive(Supplier<Boolean> liveCheck) { liveChecks.add(liveCheck); } @Override public synchronized void run() { // don't start renewing immediately try { wait(ttl / 2 * 1000); } catch (InterruptedException e) {} while (!isShutdown()) { if (liveChecks.isEmpty() || liveChecks.stream().allMatch(Supplier::get)) { client.sessionClient().renewSession(getId()); } try { wait(ttl / 2 * 1000); } catch (InterruptedException e) { // go on, try again } } } public synchronized boolean isShutdown() { return shutdown; } public synchronized void close() { shutdown = true; notify(); client.sessionClient().destroySession(getId()); } } 

Then the choice of a leader is more or less simple than:

 if (consul.keyValueClient().acquireLock(getServiceKey(service), currentNode, sessionHolder.getId())) return true; // I'm the leader 

One thing that needs to be remembered is that if the session ends without clearing (which I do above in SessionHolder.close() ), the consul's lock-delay function will prevent the selection of a new leader for about 15 seconds (by default, which unfortunately, the Consul client does not offer an API for change).

To solve this problem, in addition to making sure that the final services are cleaned after themselves, as shown above, I also guarantee that the service maintains its leadership position for the minimum time necessary to release the leadership when no longer using it, calling consul.keyValueClient().releaseLock() . For example, I have a cluster service in which we select a leader to read data updates from an external RDBMS (which are then distributed directly to the cluster, and not every node, reloading all the data). Since this is done through a survey, each node will try to be elected before the poll, and if it is elected, it will test the database, distribute the update, and resign. If after this a failure occurs, delay-lock will not prevent another node from being polled.

Answer 3

If this is still relevant, I (hopefully) bypassed the potential for false positives:

• TTL Health Check Registration
• Linking a session to this check only

Corresponding code snippet:

 sessionClient.createSession( ImmutableSession.builder() .addChecks(checkId) // Ties the session to this check .behavior("delete") .lockDelay("15s") .build() )