ENH: Add Arrow CSV Reader

[lithomas1]

• closes ENH: allow engine='pyarrow' in read_csv #23697
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

Picking up from #38370.
Perf is OK. (StringIO/Text buffer reading gets like 20% overhead b/c of BytesIOWrapper, we should probably look into more efficient ways of doing it/cythonize it. Bytes IO performance looks good though).

Benchmarks 100k rows, 5 col

[  0.00%] ··· io.csv.ReadCSVEngine.time_read_bytescsv                                                                                                                             ok
[  0.00%] ··· ========= ============
                engine              
              --------- ------------
                  c       22.9±2ms  
                python    259±20ms  
               pyarrow   5.79±0.3ms 
              ========= ============

[  0.00%] ··· io.csv.ReadCSVEngine.time_read_stringcsv                                                                                                                            ok
[  0.00%] ··· ========= ============
                engine              
              --------- ------------
                  c       20.6±1ms  
                python    257±1ms   
               pyarrow   7.06±0.3ms 
              ========= ============

Tests will still fail tho, working on that rn.

My machine is a 2019 MBP(Intel) with 6 cores. YMMV based on the number of CPUs you have. Surprised that it doesn't quite scale linearly, maybe the file is too small to feel the benefit of all 6 cores.

[twoertwein]

Picking up from #38370.
Perf is OK. (StringIO gets like 20% overhead b/c of BytesIOWrapper, we should probably look into more efficient ways of doing it/cythonize it).

Isn't the BytesIOWrapper only used when the user provides a text-handle? If the user provides something path-like, get_handle should (hopefully) directly open it in binary mode, or?

[lithomas1]

Yup, that's why the StringIO benchmark is slower(bytescsv is unaffected as you can see above).. Hopefully the bytes benchmark is closer to the true speed of pyarrow, but haven't profiled. Perf there does look reasonable to me.

Sorry for being a little ambiguous in the description.

[lithomas1]

I think I've addressed all the comments on the previous PR. LMK otherwise. This is ready for a preliminary review now, will ping everyone involved once I get CI to green.(22 tests to go left, but some tests are getting stuck and timing out).

[lithomas1]

as green as it gets.

[lithomas1]

also cc @arw2019 @xhochy

[jreback]

looks really good, thanks @lithomas1 for reviving this.

some doc-comments / requests
and 1 for code.

are all of the currently raised exceptions tested?

[lithomas1]

Raised exceptions tested here.
https://github.com/pandas-dev/pandas/pull/43072/files#diff-2a0f93df26fa0ee1d10f216142db735ff227fcfb02f161ad165fdfcb3f767cd1R125-R146.

[jreback]

@pandas-dev/pandas-core if any comments. Ideally we could do these in a followup.

[Dr-Irv]

Some minor comments

[lithomas1]

Ok, I think we should try to merge this now, if no other comments(it has been 1 week). Planning on enabling more features and doing some parser cleanup after this.

[jreback]

thanks @lithomas1 very nice!

pls create a follow up issue with checkboxes

[jreback]

thanks @arw2019 for a lot of the initial work here!

[lithomas1]

#38889
#38872

FWIW, is it possible to create an Arrow label for pyarrow things? Arrow issues are still filed under IO Parquet which is wrong. I think this might be relevant for things like the ArrowStringArray too.

[jreback]

i created a label just now

[jbrockmendel]

im seeing pyarrow io tests stalling out locally (and not KeyboardInterrupt-able), can anyone else on mac confirm?

[lithomas1]

Maybe try setting the env var OMP_NUM_THREADS to 1? We should be running these tests single threaded right now.
I'll try to reproduce when I get back on my mac, but its interesting how this is not failing on CI.

[lithomas1]

Ok, looking into this right now. I think I can reproduce locally, but interested in how this doesn't reproduce on CI.

[jbrockmendel]

I fixed it locally by upgrading pyarrow