Fixed LOAD DATA LOCAL INFILE commands

[srstrickland]

• Fixes for both "buffered" & "unbuffered" cursor types
• Registered test_load_local to run with tests
• Refactored test_load_local tests to work with the tornado framework

More information here:
#29

[methane]

In LoadLocal.send_data:

                     chunk = open_file.read(chunk_size)

This line will block entire your ioloop.

[methane]

In same function:

                     self.connection._write_bytes(packet)

This only append packet into write buffer.
You should wait here.

[methane]

As you can see, writing proper async code is hard job.
Wrong code works for simple case, including most unit test.
This is why I can't recommend this project easily, and I want to minimize this library.

[srstrickland]

Fair enough. My initial goal was just to get it functional, not necessarily to vet all the "async" portions of it. But I've taken your suggestions & revised. You can obviously take it or leave it. Thanks

[methane]

FYI, current chunk_size is too large. 2**24 may be larger than OS readahead size.
I think 16KB is optimal.
While waiting data sent to socket, OS fills readahead buffer background. If next read is smaller than
filled readahead buffer, it can be nonblocking.

[srstrickland]

True... but in reality the stream is going to be doing nonblocking reads against the file descriptor (PipeIOStream sets O_NONBLOCK)... and no OS is going to give back that much data in a single nonblocking read, which governs the chunk size that gets handled. That said, it probably would be better to take control of that more tightly.

[methane]

I can't understand what stream you mean.
Your code read from file (not IOStream) and write it to stream (TCP or Unix socket stream).
Where did you use PipeIOStream??

[methane]

See also: go-sql-driver/mysql#424

[srstrickland]

After you made the comment about the blocking file read, I changed to use tornado.iostream.PipeIOStream:

https://github.com/Tubular/Tornado-MySQL/blob/fix/load_data_local/tornado_mysql/connections.py#L1253-1254

[methane]

If you want to use PipeIOStream, you should use pipe. Maybe, "/usr/bin/cat /path/to/file" (This is not works on Windows, off course).

File I/O can't be non blocking. That's why libuv uses threadpool for file I/O.

[srstrickland]

Yes, my mistake. The truth is that for my application it doesn't matter much, since the portions that use LOAD DATA INFILE are not time critical and are not simultaneously serving, say, user requests. I had initially written a version that uses a ThreadPoolExecutor behind the scenes to do the file reads, but for python27 that would've required the futures pip package (native in py3). So I punted on that for now. I see that you removed infile support, so I'm assuming you won't want to incorporate my changes, even if I do get nonblocking reads in. Ping me if you ever change your mind.

[methane]

FYI, aio-libs/aiomysql is actively maintained. Tornado can be used with asyncio now.
I recommend you to use Python 3.5 and aiomysql, not Python 2.7 and this.

[srstrickland]

That's the eventual plan, actually... but for now we have to support py2.7 & py3.4.