read_csv: usecols doesn't work if separator is not ","

[bmu]

If I have data (without a header) separated by "," usecols works, however if the data is separated by white space it doesn't seem to work:

In [30]: data = '1,2,3\n4,5,6\n7,8,9'

In [31]: pd.read_csv(StringIO(data), usecols=[0, 1], header=None)
Out[31]: 
   0  1
0  1  2
1  4  5
2  7  8

In [32]: data = '1 2 3\n4 5 6\n7 8 9'

In [33]: pd.read_csv(StringIO(data), sep='\s+', header=None)
Out[33]: 
   0  1  2
0  1  2  3
1  4  5  6
2  7  8  9

In [34]: pd.read_csv(StringIO(data), sep='\s+', usecols=[0, 1], header=None)
Out[34]: 
   0  1  2
0  1  2  3
1  4  5  6
2  7  8  9

[garaud]

I had a strange problem with the parameter usecols yesterday -- with a ';' separator. I'll try to give an example. I didn't get that it might be a separator problem.

[garaud]

I wrote a new test case in test_parsers:TestCParserLowMemory. This test fails according to the problem.

See garaud@fe36220

I don't get what it's wrong. I took a look on #2654. I think I'll read pandas/src/parser.pyx.

[wesm]

The C parser does not support multi-character and regex delimiters yet. Try delim_whitespace=True. The pure Python parser does not have usecols implemented but this shouldn't be too difficult to do

[garaud]

OK. Thanks ! Good to know. Fisrt, I'll try with the delim_whitespace parameter. Then, I'll propose a "usecols" implementation in the pure Python parser (with its test).

Edit:

In [24]: pd.read_csv(StringIO(data), delim_whitespace=True, header=None, usecols=(1,2))
Out[24]: 
   1  2
0  2  3
1  5  6
2  8  9

works fine.

[garaud]

See #2748

A test case with the delim_whitespace parameter.

[wesm]

i added an explicit error message for this case. pushing fixing it til later

[dsm054]

Is this still an issue?

>>> import pandas as pd
>>> from StringIO import StringIO
>>> pd.__version__
'0.13.1-343-g6efa4c1'
>>> data = '1 2 3\n4 5 6\n7 8 9'
>>> pd.read_csv(StringIO(data), sep='\s+', usecols=[0, 1], header=None)
   0  1
0  1  2
1  4  5
2  7  8

[3 rows x 2 columns]
>>> pd.read_csv(StringIO(data), sep='\s+', usecols=[0, 1], header=None, engine='c')
   0  1
0  1  2
1  4  5
2  7  8

[3 rows x 2 columns]
>>> pd.read_csv(StringIO(data), sep='\s+', usecols=[0, 1], header=None, engine='python') 
   0  1
0  1  2
1  4  5
2  7  8

[3 rows x 2 columns]
>>> pd.read_csv(StringIO(data.replace(" ",";")), sep=';', usecols=[1, 2], header=None, engine='c')
   1  2
0  2  3
1  5  6
2  8  9

[3 rows x 2 columns]
>>> pd.read_csv(StringIO(data.replace(" ",";")), sep=';', usecols=[1, 2], header=None, engine='python')
   1  2
0  2  3
1  5  6
2  8  9

[3 rows x 2 columns]

[jreback]

hmm seems ok

can u take a quick look and see if u can find a release note that fixed this?

[dsm054]

Maybe it's #5211?

[jreback]

closed by #5211