Graceful shutdown

[DifferentialOrange]

If connected to Tarantool 2.10 or newer, after this patch a connection supports server graceful shutdown [1]. In this case, server will wait until all client requests will be finished and client disconnects before going down (server also may go down by timeout). Client reconnect will happen if connection options enable reconnect.

1. https://www.tarantool.io/en/doc/latest/dev_guide/internals/iproto/graceful_shutdown/

I didn't forget about (remove if it is not applicable):

• Tests (see documentation for a testing package)
• Changelog (see documentation for changelog format)
• Documentation (see documentation for documentation style guide)

Closes #214

[DifferentialOrange]

For now I've got an issue that not all requests started with Do are ensured to get executed. It's more like a Tarantool behavior rather than we did something wrong.

[DifferentialOrange]

Well, it seems that it is still not working: I've got TestGracefulShutdownRespectsClose dead stuck once and pool tests are stuck too.

[DifferentialOrange]

Well, it seems that it is still not working: I've got TestGracefulShutdownRespectsClose dead stuck once and pool tests are stuck too.

Seems like the issue was again an concurrent Unregister in connectionClose and shutdown. Started to Unregister in separate goroutine, this should solve the issue.

[DifferentialOrange]

For now I've got an issue that not all requests started with Do are ensured to get executed. It's more like a Tarantool behavior rather than we did something wrong.

Added simple notify channel to track shutdown callback events if one wish to do so.

[DifferentialOrange]

There still seems to be some rare concurrency issues (with shutdown notify channel)

[DifferentialOrange]

I've re-checked one more time and we definitely cannot replace inst.Cmd.Process.Signal(syscall.SIGTERM) with inst.Cmd.Process.Kill() since it sends SIGKILL which kills a Tarantool mercilessly without allowing to handle shutdown triggers.

[DifferentialOrange]

The last known bug related to

--- FAIL: TestGracefulShutdownConcurrent (1.15s)
    shutdown_test.go:541: 
        	Error Trace:	shutdown_test.go:541
        	Error:      	Expected nil, but got: tarantool.ClientError{Code:0x4001, Msg:"connection closed after server shutdown"}
        	Test:       	TestGracefulShutdownConcurrent
        	Messages:   	No errors on concurrent wait

was caused by if conn.shutdownWatcher != nil { conditions: shutdown watcher requests itself could be sent BEFORE conn.shutdownWatcher is set, so counter is not incremented, but may finish AFTER conn.shutdownWatcher is set, so counter will be decremented and became negative. I don't think it's a big perf issue so I left them unconditional.

[DifferentialOrange]

So it seems that everything we wanted to solve is solved now.

[oleg-jukovec]

LGTM, but there are a couple of comments that need to be resolved.

[oleg-jukovec]

I rebased the branch because I want to merge the pull request today. I also removed the code from connection.go:

    // future here does not belong to any shard yet,
    // so cancelFuture don't call markDone.
    conn.decrementRequestCnt()

because a future actually belongs to a shard after successful newFuture().