Jersey Server-Sent Events - writing to a broken connection does not throw an exception

Question

We use Jersey Server Events (SSE) to allow remote components of our application to listen for events raised by our Jersey / Tomcat server. This works great.

However, it is very important that our server has an accurate list of listeners currently connected (our remote components). To this end, our server sends a tiny message to each caller (via eventOutput.write) once every five seconds. If our remote component is disconnected while connecting to the SSE or if the remote computer is disconnected while connecting to the SSE, our eventOutput.write server throws a ClientAbortException / SocketException exception, shown below. This is great: we will catch the exception, note that the caller is no longer connected, and move on.

Now, for the problem. As I already mentioned, eventOutput.write throws an exception when our remote component software does not work or where the computer on which it works is turned off. However, there are two cases where the eventOutput.write call on a computer with a longer connection does NOT throw an exception: 1) if the Ethernet cable of the remote computer is simply pulled out when the caller is connected to the SSE, and 2) if the network adapter on the remote computer is turned off (i.e. e. by administrative action), while the caller is connected to the SSE. In these two cases, we can call eventOutput.write on the remote computer every five seconds for several hours and not raise an exception. This makes it impossible to detect that the remote computer is no longer connected.

I see that EventOutput (and ChunkedOutput) has very few methods and properties, but I'm wondering if there is a way to configure or use it, which will cause the exception to be sent when writing to a remote computer that was disconnected with the Ethernet cable disconnected or disconnecting the network adapter.

And here is the (good / useful) exception that we get when eventOutput.write MAKES the exception we want:

org.apache.catalina.connector.ClientAbortException: null at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371) ~[catalina.jar:7.0.53] at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333) ~[catalina.jar:7.0.53] at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101) ~[catalina.jar:7.0.53] at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.flush(ResponseWriter.java:303) ~[jaxrs-ri-2.13.jar:2.13.] at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:292) ~[jaxrs-ri-2.13.jar:2.13.] at org.glassfish.jersey.server.ChunkedOutput$1.call(ChunkedOutput.java:240) ~[jaxrs-ri-2.13.jar:2.13.] at org.glassfish.jersey.server.ChunkedOutput$1.call(ChunkedOutput.java:190) ~[jaxrs-ri-2.13.jar:2.13.] at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jaxrs-ri-2.13.jar:2.13.] at org.glassfish.jersey.internal.Errors.process(Errors.java:242) ~[jaxrs-ri-2.13.jar:2.13.] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:347) ~[jaxrs-ri-2.13.jar:2.13.] at org.glassfish.jersey.server.ChunkedOutput.flushQueue(ChunkedOutput.java:190) ~[jaxrs-ri-2.13.jar:2.13.] at org.glassfish.jersey.server.ChunkedOutput.write(ChunkedOutput.java:180) ~[jaxrs-ri-2.13.jar:2.13.] at com.appserver.webservice.AgentSsePollingManager$ConnectionChecker.run(AgentSsePollingManager.java:174) ~[AgentSsePollingManager$ConnectionChecker.class:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_71] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_71] at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[na:1.7.0_71] at java.net.SocketOutputStream.write(SocketOutputStream.java:159) ~[na:1.7.0_71] at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215) ~[tomcat-coyote.jar:7.0.53] at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) ~[tomcat-coyote.jar:7.0.53] at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119) ~[tomcat-coyote.jar:7.0.53] at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799) ~[tomcat-coyote.jar:7.0.53] at org.apache.coyote.Response.action(Response.java:174) ~[tomcat-coyote.jar:7.0.53] at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366) ~[catalina.jar:7.0.53] ... 19 common frames omitted 

Answer 1

I don’t think that it will be possible to take into account all possible failures by adding code around SSE sockets, even if Jersey adds all the information available from the socket interface. The only viable solution is the right two-way communication. In the case of SSE streaming, a pulled cable does not cause an interruption, because nothing should tell it that the remote host is now unavailable (until the OS closes the socket).

Your first step is right - implement heartbeats every N seconds. Then all you have to do is report back with another tiny http call so often that you are still listening. It is up to you to confirm every 5 seconds or every minute - it depends on how quickly you need to detect the problem.

You can do this in the same Jersey resource by doing @POST (in RESTful terms, he reads “create a new account in which you receive events”).

Note: browsers reconnect SSEs well in case of network interruptions, no need to bother with it.