Add integration tests against throwaway mysql instance

[mherr-google]

I would like to contribute integration tests that test the behavior of the driver against various connectivity corner cases. To start with, these are:

1. client sends query, but server kills connection before response received
2. server kills connection while client is idle, client tries to reuse connection.

These will be done inside a Go binary that launches a throwaway instance of mysqld, interacts with it using the driver and ensures that certain error responses are returned.

To avoid checking any MySQL binaries into the repo, the binary will take some arguments (eg. --mysqld=/usr/bin/mysqld) which will be used to locate the binary launched. This means it will not automatically run as a unit test, but it should still be straightforward to run when making changes.

My intention is that we can steadily grow this test and use it to objectively demonstrate or refute connectivity issues -- pull requests to fix an edge case could be required to add a testcase to this file along with the change, avoiding back-and-forth over issues.

They can also be used as living documentation of the driver's behavior under specific conditions.

Please let me know if you have any concerns -- I plan to start working on this shortly.

[mherr-google]

Could you please assign this issue to me if you are happy for me to work on this?

Thanks

[mherr-google]

(Actually, I'll add these to driver_test.go, which looks like it has everything I need)

[methane]

These will be done inside a Go binary that launches a throwaway instance of mysqld, interacts with it using the driver and ensures that certain error responses are returned.

I'm not sure, but it seems fragile test to me. How can you control timing precisely?

End-to-end test seems nice for users who has own MySQL environment.

Timing corner case test is also nice. But I think dummy MySQL server implemented in Go is better for it.

[mherr-google]

I'm not sure, but it seems fragile test to me. How can you control timing precisely?
We'd have to use a variety of tricks similar to those used to test multithreaded code.

For example, to check cancelling a running query, we need to ensure we don't cancel until the query is running, otherwise the test is bogus. We can do this by asking the server to execute multiple statements inside a single Exec, and look for the side-effect of the first statement by polling the database. The second statement could be a "SELECT SLEEP(1000000)" to effectively block forever.

We can use similar strategies to test other kinds of conditions. I noticed while writing these tests that the driver behaves very differently for "unix" and "tcp" dialers. This is because a "unix" socket immediately breaks with "broken pipe" when a mysql "KILL" statement is run, but a "tcp" socket will not. This is a Linux-implementation detail: the Linux TCP stack doesn't cause write(2) to fail when it has a received a RST or FIN packet for a socket; it simply appends to the socket buffer. However, the "unix" socket close call synchronously breaks the pipe -- after the call returns, any write will immediately fail. Having the tests exercise these surprising behaviors in code might save time explaining them to newcomers.

I've also had success locally with a small tcp proxy placed in front of mysqld that I can use to deliberately break, delay connections or, say, simulate server unresponsiveness. That's a simple Go class we could instantiate inside tests pretty cheaply.

[dolmen]

I suggest to implement that in the Travis-CI environment to make those tests available to any contributors without requiring a local MySQL server instance.

[julienschmidt]

Generally I'd prefer to simulate these corner cases against a mocked server rather than against some actual MySQL instances. Especially timing tests tend to be flaky.

If we really need throwaway instances: Docker. Travis CI runs everything that runs in docker, and there are attempts to make our tests easier accessible through docker-compose anyway: #697.