move io.c from using unbuffered fread()s to read()s.

[dgwynne]

pandas already buffers reads coming from io.c itself, so it previously
used setbuf() to disable buffering inside fread(). however, certain
implementations of unbuffered stdio reads are sub-optimal. for example,
fread() in solaris ends up doing a read() for each individual byte of
the underlying filedescriptor, which turns out to be very slow.

instead, this code now open()s a file descritor and read()s directly
into the buffer that pandas has already allocated. this is effectively
what other libcs (eg, glibc) do underneath an unbuffered fread() anyway,
but this is more explicit.

while here, this tweaks the mmap backend to use open() too, and also
properly checks for mmap failure by comparing its result to MAP_FAILED
instead of NULL.

• closes #xxxx
• tests added / passed
• passes git diff upstream/master --name-only -- '*.py' | flake8 --diff
• whatsnew entry

[jreback]

https://travis-ci.org/pandas-dev/pandas/jobs/222988790
couple of cpplint errors.

Is there any kind of test which can detect this? (or just perf on some systems)?

[dgwynne]

As you implied, it isn't a hard failure. The problem is performance on Solaris libc based systems is bad. A test of the bad performance is simply to use read_csv() on such a system and compare the runtime before and after the change.

[codecov]

Codecov Report

Merging #16039 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #16039      +/-   ##
==========================================
+ Coverage   90.99%      91%   +<.01%     
==========================================
  Files         153      153              
  Lines       50481    50481              
==========================================
+ Hits        45937    45938       +1     
+ Misses       4544     4543       -1

Flag	Coverage Δ
#multiple	88.76% <ø> (ø)	⬆️
#single	40.43% <ø> (ø)	⬆️

Impacted Files	Coverage Δ
pandas/core/common.py	90.68% <0%> (-0.35%)	⬇️
pandas/util/testing.py	80.73% <0%> (+0.18%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd1031f...b36c1ff. Read the comment docs.

[codecov]

Codecov Report

Merging #16039 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #16039      +/-   ##
==========================================
- Coverage   90.84%   90.82%   -0.03%     
==========================================
  Files         159      159              
  Lines       50791    50791              
==========================================
- Hits        46143    46132      -11     
- Misses       4648     4659      +11

Flag	Coverage Δ
#multiple	88.6% <ø> (-0.03%)	⬇️
#single	40.36% <ø> (ø)	⬆️

Impacted Files	Coverage Δ
pandas/plotting/_converter.py	63.54% <0%> (-1.82%)	⬇️
pandas/core/common.py	91.03% <0%> (+0.34%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b17e286...b29b032. Read the comment docs.

[dgwynne]

I'm not sure, but I don't think the new build failures are the result of b36c1ff, which was basically whitespace tweaks. Alternatively fixing lint has made a later check run which did hit a problem. However, I don't think that is the case here.

[jreback]

can you show running an asv (perf testing) on csv benchmarks (just to assert that nothing has changed).

[dgwynne]

ill try and figure out asv. in the meantime, here are some runs before and after the change.

the py code i tested with:

import pandas as pd

data = pd.read_csv('yellow_tripdata_2016-12.csv')
print('Length: {} rows.'.format(len(data)))

this is a zone (container) on a solaris derivative box (https://www.joyent.com/smartos):

simple wc (to warm the caches?):

[admin@ade240c5-d3bf-40b3-8b24-1940e686775a ~]$ time wc -l yellow_tripdata_2016-12.csv 
10449409 yellow_tripdata_2016-12.csv

real    0m0.973s
user    0m0.600s
sys     0m0.371s

before:

[admin@ade240c5-d3bf-40b3-8b24-1940e686775a ~]$ time python pd.py 
Length: 10449408 rows.

real    35m8.845s
user    3m24.646s
sys     31m42.769s

after:

[admin@ade240c5-d3bf-40b3-8b24-1940e686775a ~]$ time python pd.py 
Length: 10449408 rows.

real    0m36.593s
user    0m30.888s
sys     0m4.038s

pretty brutal.

on a linux box:

wc:

uqdgwynn@mango ~$ time wc -l yellow_tripdata_2016-12.csv 
10449409 yellow_tripdata_2016-12.csv

real    0m0.945s
user    0m0.409s
sys     0m0.517s

before:

uqdgwynn@mango ~$ time python pd.py 
Length: 10449408 rows.

real    1m25.830s
user    0m56.329s
sys     0m26.963s
uqdgwynn@mango ~$ time python pd.py 
Length: 10449408 rows.

real    1m17.969s
user    1m2.792s
sys     0m14.248s

after:

uqdgwynn@mango ~$ time python pd.py 
Length: 10449408 rows.

real    0m47.950s
user    0m37.004s
sys     0m8.825s

im surprised the linux test shows a difference.

[dgwynne]

i ran asv continuous -f 1.1 -E virtualenv origin/parser-no-stdio HEAD in the asv_bench directory, and after each individual benchmark it produced "BENCHMARKS NOT SIGNIFICANTLY CHANGED." in green.

[jreback]

this looks good perf wise (neutral on my tests), so if its better on other platforms +1.

[jreback]

thanks @dgwynne