Speeding up the shutdown of the driver.

[PetrJanouch]

No description provided.

[pstutz]

Thanks for looking into this, this helps with our issue #571.

With a quiet period of 200ms the shutdown still takes about 1 second.
With a not-so-graceful quiet period of 0ms the shutdown is almost instantly.

What important work wouldn't get done if the shutdown wasn't graceful? Might it make sense to either set it to 0ms or to offer a shudownNow() method for users that can live with a possible drawback?

[PetrJanouch]

@pstutz Unfortunately, we cannot set the quiet period to 0ms right now. It seems that there are code paths that can result to some close or clean up tasks being submitted to the event loop after shutdown is called on the event loop. I think the ideal solution is to make sure that calling shutdown on the event loop is really the absolutely last step in the shut down routine. We could set the quiet period to 0ms then. Unfortunately, this is not as easy as it sounds, because of the asynchronous and event driven nature of that code. It will require revisiting and carefully ordering the shutdown logic. We will see what we can do. shutdownNow() might result in some resources not being closed (the closing events not being processed by the event loop)

[technige]

What does this do? Add a timeout? What happens if the timeout expires... what is the user supposed to do?

I am very cautious about optimising for a very niche use case. We need to optimise for the general case, wherein the driver shuts down at application shutdown. Here, everything must close; there is no other code path.

[PetrJanouch]

A short explanation how the shutdownGracefully works. It has 2 parameters ‘quiet period’ and ‘timeout’. The algorithm that it uses is the following:

shutdownStart = now()
while (true) {
	Thread.sleep(quietPeriod)
	if (nothingWasSubmittedToTheEventLoopWhileSleeping()) {
           break
	}

	if (now() - shutdownStart > timeout) {
          // timeout
	  break
        }
}

The main purpose of this shutdown algorithm is to compete work that is in progress.
The default value for ‘quiet period’ are 2 seconds and 15 seconds for ‘timeout’

What do we need a quiet period for anyway? The current shutdown logic is like this:

channels.each (ch -> closeAsync())
shutdownEventLoopWithQuietPeriod()

The graceful shutdown with the quiet period is needed because closing a channel async generates events submitted to the event loop.

What we would need is something like this:

channels.each (ch -> closeAsync())
waitForAllChannelsToBeClosed()
shutdownEventLoopWithoutQuietPeriod()

As simple as the pseudo code above looks, the problem is that achieving waitForAllChannelsToBeClosed() with current Netty API is not that simple (at least was not the last time I checked)

The problem is that the shutdown currently takes 4s (the quiet period is usually reset once by the tasks generated by closing the channels) which is enough to make for instance lambda users or users with a lot of integration tests sad.

The reasoning behind my PR was that the defaults in Netty are set so it would work with all possible use cases the toolkit is used for, but 200ms quiet period is more that enough for us.