Tornado Graceful Closure IOLoop

Question

I use the following code to gracefully close the tornado application (taken from https://gist.github.com/wonderbeyond/d38cd85243befe863cdde54b84505784 ):

def sig_handler(servers, sig, frame): io_loop = tornado.ioloop.IOLoop.instance() def stop_loop(deadline): now = time.time() if now < deadline and (io_loop._callbacks or io_loop._timeouts): logging.info('Waiting for next tick') print("CALL BACKS") print(io_loop._callbacks) print("TIMEOUTS") print(io_loop._timeouts) io_loop.add_timeout(now + 1, stop_loop, deadline) else: io_loop.stop() logging.info("Shutting down.") def shutdown(): logging.info("Stopping http servers") # servers is a list of servers to stop for s in servers: s.stop() logging.info("Will shutdown in %s seconds ...", MAX_WAIT_SEC_BEFORE_SHUTDOWN) stop_loop(time.time() + MAX_WAIT_SEC_BEFORE_SHUTDOWN) logging.warning("Caught signal: %s", sig) io_loop.add_callback_from_signal(shutdown) 

I set MAX_WAIT_SEC_BEFORE_SHUTDOWN for 10 seconds. Even after closing the http servers, it takes only 10 seconds to close the server each time. I noted that there are always elements in the io_loop._timeouts list For example:

 [<tornado.ioloop._Timeout object at 0x106b90408>, <tornado.ioloop._Timeout object at 0x106b904c8>, ...] 

What are the elements in io_loop._timeouts ? Should I expect this to be an empty list, or am I not stopping something that I should have?

Is this normal operation a shutdown? Can anyone suggest another code?

Answer 1

Stopping the HTTPServer tornado does not close all existing connections, but only stops the server from accepting new ones. There is an undocumented close_all_connections method , but it does not distinguish between unoccupied connections and those that are currently processing requests, and therefore not suitable for use in graceful disconnection. There is currently no way to close all connections as they are idle.

Each idle connection maintains a timeout on IOLoop (the connection will be closed if it takes too long, although the default value is one hour. Go idle_connection_timeout= to the HTTPServer constructor to change this). In addition, other functions of Tornado or other libraries can also create timeouts (in Tornado itself, curl_httpclient and autoreload both create timeouts that always work), so you can’t even assume that the number of pending timeouts will ever reach zero .

In general, I recommend just unconditionally waiting for MAX_WAIT_SEC_BEFORE_SHUTDOWN instead of trying to determine whether it is safe to close earlier. If you want to determine when all pending operations are completed, then it is best to count pending operations (for any definition of “operation” that you find appropriate) instead of trying to do this from the details of the IOLoop implementation.