LOAD LOCAL can send file larger than max_allowed_packet

[methane]

Description

"LOAD LOCAL INFILE" query can send file larger than max_allowed_packet. But go-sql-driver/mysql returns error when file is large.

http://dba.stackexchange.com/questions/13160/does-the-max-allowed-packet-variable-have-any-effect-on-load-local-infile

Additionally, some user set max_allowed_packet very large (e.g. 1GB).
Since max size of MySQL packet is 16MB, using it for buffer size is not good.
Typical readahead buffer size of Linux is 128KB. So large buffer size may cause performance penalty. (#364).
I choose 16KB for buffer size (default of net_buffer_length, http://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_net_buffer_length).

Checklist

• Code compiles correctly
• Created tests, if possible
• All tests passing
• Extended the README / documentation, if necessary
• Added myself / the copyright holder to the AUTHORS file

[arnehormann]

@methane I like this - thank you! Looks like very sound reasoning.
I'm very curious/optimistic regarding the performance.
Can you add a benchmark and post numbers?

[methane]

Benchmarking this in real world is bit hard. It's heavily rely on slow disk read and stable server performance. I don't have good environment to test this for now.

But I can emulate in toy program. loadfile.go

inada-n@nico:~$ dd if=/dev/zero of=blob bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB) copied, 1.96072 s, 68.5 MB/s
inada-n@nico:~$ ll blob
-rw-rw-r-- 1 inada-n inada-n 134217728 Feb 24 18:34 blob
inada-n@nico:~$ echo 1 | sudo tee /proc/sys/vm/drop_caches
1
inada-n@nico:~$ time ./loadfile 8k blob
2016/02/24 18:34:31 block size = 8192

real    0m45.668s
user    0m0.268s
sys 0m3.532s
inada-n@nico:~$ echo 1 | sudo tee /proc/sys/vm/drop_caches
1
inada-n@nico:~$ time ./loadfile 16k blob
2016/02/24 18:35:33 block size = 16384

real    0m16.554s
user    0m0.384s
sys 0m1.020s
inada-n@nico:~$
inada-n@nico:~$ echo 1 | sudo tee /proc/sys/vm/drop_caches
1
inada-n@nico:~$ time ./loadfile 32k blob
2016/02/24 18:36:00 block size = 32768

real    0m15.634s
user    0m0.100s
sys 0m0.828s
inada-n@nico:~$ echo 1 | sudo tee /proc/sys/vm/drop_caches
1
inada-n@nico:~$ time ./loadfile 64k blob
2016/02/24 18:36:29 block size = 65536

real    0m14.762s
user    0m0.000s
sys 0m0.668s
inada-n@nico:~$ echo 1 | sudo tee /proc/sys/vm/drop_caches
1
inada-n@nico:~$ time ./loadfile 128k blob
2016/02/24 18:36:55 block size = 131072

real    0m14.002s
user    0m0.000s
sys 0m0.376s
inada-n@nico:~$ echo 1 | sudo tee /proc/sys/vm/drop_caches
1
inada-n@nico:~$ time ./loadfile 256k blob
2016/02/24 18:37:25 block size = 262144

real    0m15.898s
user    0m0.028s
sys 0m0.444s
inada-n@nico:~$ echo 1 | sudo tee /proc/sys/vm/drop_caches
1
inada-n@nico:~$ time ./loadfile 16m blob
2016/02/24 18:37:59 block size = 16777216

real    0m15.723s
user    0m0.000s
sys 0m0.220s

dstat for 16mb show iowait:

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  0   0 100   0   0   0|   0     0 | 684B  860B|   0     0 |  21    40
  0   1  83  16   0   0|  16M    0 | 648B  844B|   0     0 | 357   453
  0   0  99   1   0   0|2176k    0 | 108B  228B|   0     0 |  53    63
  0   1  88  11   0   1|  16M    0 | 108B  436B|   0     0 | 174   299
  0   0 100   0   0   0|   0    16k| 108B  436B|   0     0 |  12    12
  0   0  88  12   0   0|  16M    0 | 108B  436B|   0     0 | 190   380
  0   0 100   0   0   0|   0     0 | 108B  436B|   0     0 |  15    17
  0   1  88  12   0   0|  16M    0 | 108B  436B|   0     0 | 194   381
  0   0 100   0   0   0|   0     0 | 108B  436B|   0     0 |   9    10
  0   0  89  11   0   0|  16M    0 | 108B  436B|   0     0 | 196   389
  0   0 100   0   0   0|   0     0 | 108B  436B|   0     0 |  11    16
  0   1  83  16   0   0|  16M    0 | 108B  436B|   0     0 | 199   386
  0   0 100   0   0   0|   0     0 | 108B  436B|   0     0 |  12    19
  0   1  86  13   0   0|  16M    0 | 108B  436B|   0     0 | 330   392
  0   0 100   0   0   0|   0     0 | 108B  436B|   0     0 |  12    18
  0   1  88  12   0   0|  16M    0 | 108B  436B|   0     0 | 323   377
  0   0 100   0   0   0|   0     0 | 108B  436B|   0     0 |  10    15
  0   0 100   0   0   0|   0     0 | 216B  872B|   0     0 |  21    79

There are no iowait in dstat for 16KB:

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  0   0 100   0   0   0|   0     0 | 998B  886B|   0     0 |  27    54
  0   0 100   0   0   0|   0     0 | 396B  648B|   0     0 |  22    32
  0   1  95   5   0   0|3116k    0 | 360B  632B|   0     0 | 400   821
  0   0 100   0   0   0|7936k    0 | 108B  574B|   0     0 |1537  6273
  1   0  98   1   0   0|7936k   24k| 270B   90B|   0     0 |1567  6473
  0   0 100   0   0   0|8064k    0 | 108B 1186B|   0     0 |1567  6465
  0   1  99   0   0   0|7936k    0 | 108B  436B|   0     0 |1576  6479
  0   0 100   0   0   0|7936k    0 | 108B  436B|   0     0 |1581  6486
  0   0 100   0   0   0|8064k    0 | 108B  436B|   0     0 |1560  6479
  0   0 100   0   0   0|7936k    0 | 108B  436B|   0     0 |1573  6496
  0   1  99   0   0   0|7936k    0 | 108B  436B|   0     0 |1562  6465
  0   0 100   0   0   0|8064k    0 | 108B  436B|   0     0 |1569  6484
  0   0 100   0   0   0|7936k    0 | 108B  436B|   0     0 |1558  6479
  0   0 100   0   0   0|8064k    0 | 108B  436B|   0     0 |1579  6501
  0   0 100   0   0   0|7936k    0 | 108B  436B|   0     0 |1564  6463
  1   0  99   0   0   0|7936k    0 | 108B  436B|   0     0 |1575  6472
  0   0 100   0   0   0|8064k    0 | 108B  436B|   0     0 |1579  6495
  0   0 100   0   0   0|7936k    0 | 108B  436B|   0     0 |1570  6496
  0   0 100   0   0   0|8064k    0 | 108B  436B|   0     0 |1572  6474
  0   0 100   0   0   0|1824k    0 | 216B  872B|   0     0 | 419  1726
  0   0 100   0   0   0|   0     0 | 108B  276B|   0     0 |  11    12
  0   0 100   0   0   0|   0     0 | 108B  436B|   0     0 |  14    18

I ran this benchmark on VirtualBox. Small read may be faster on real environment since time.Sleep() is not so accurate.

[methane]

I don't know about how MySQL server handle "LOAD LOCAL INFILE".
But similar iowait can be happen on server side.

[arnehormann]

That's good enough for me - and reproducible by whoever wants to do so.
Thanks for the numbers, LGTM!