BUG: Patch missing data handling with usecols

[gfyoung]

Patch handling of cases when the first row of a CSV is incomplete and usecols is specified.

Closes #6710.
Closes #8985.

xref #14782.

[codecov-io]

Current coverage is 84.75% (diff: 100%)

Merging #15066 into master will decrease coverage by <.01%

@@             master     #15066   diff @@
==========================================
  Files           145        145          
  Lines         51220      51220          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43415      43413     -2   
- Misses         7805       7807     +2   
  Partials          0          0          

Powered by Codecov. Last update c71f214...6bb558d

[jreback]

@jorisvandenbossche can you have a look.

[jorisvandenbossche]

Looks good, and no concern regarding fix/test for #6710.

For issue #8985, I am a bit less sure this is necessarily the right thing. In any case, it is nowhere mentioned that usecols can actually be used for this case. So if we fix this 'bug', I would rather regard it as an enhancement to read in malformed files, and also document this.

[jorisvandenbossche]

Can you put this in a separate test that is called differently? As AFAIK this one is not about an incomplete first row (multiple rows are shorter, so it is inferred that there are 3 columns) But it is rather about ignoring excessive values on certain columns when using usecols?

[gfyoung]

Fair enough. Done.

[jorisvandenbossche]

Regarding the 'malformed' lines (rows with too many values), we eg also have error_bad_lines and warn_bad_lines. Although this will 'drop' the full bad line instead of dropping only the excessive values on the line.
Slightly related, there is also #9729 which asks for similar handling for bad lines with too few values. So maybe we should have a more general discussion about how to handle bad lines (where the options are raise/drop full line/'process' combined with a warning or not in case of not raising)

[gfyoung]

@jorisvandenbossche : Let me address your comments separately:

1. csv_reader with limited number of columns should should completely disregard the unused fields #8985 is a bug, not an enhancement. The usecols parameter was 100% valid in the test case presented from the issue. I fail to see what new thing needs to be documented here.

2. "Bad" lines with too few fields #9729 is tangentially related to this issue because it has to deal with an inconsistent number of fields across rows. usecols should not be involved in that discussion. I don't believe this discussion needs include this PR.

[jorisvandenbossche]

#8985 is a bug, not an enhancement. The usecols parameter was 100% valid in the test case presented from the issue. I fail to see what new thing needs to be documented here.

Can you point me to documentation and/or tests that specifies this behaviour of usecols ?

[gfyoung]

The usecols parameter is specifying three columns. Every row has three columns, so the parameter is perfectly valid. We should be able to handle it just fine.

[jorisvandenbossche]

We should be able to handle it just fine.

I am not saying that we necessarily shouldn't do this (the use case is certainly useful). My request above was just, if we fix this, to document this as a feature of usecols

[jorisvandenbossche]

BTW, the example case for #8985 actually already seems to work on latest master?

In [21]: usecols = [0, 1, 2]

In [22]: data = '19,29,39\n' * 2 + '10,20,30,40'

In [25]: pd.read_csv(StringIO(data), header=None, usecols=usecols)
Out[25]: 
    0   1   2
0  19  29  39
1  19  29  39
2  10  20  30

In [26]: pd.read_csv(StringIO(data), header=None, usecols=usecols, engine='python')
Out[26]: 
    0   1   2
0  19  29  39
1  19  29  39
2  10  20  30

In [27]: pd.__version__
Out[27]: '0.19.0+317.g41170d4'

[gfyoung]

Handling missing data is independent of usecols IMO. That's why this isn't a "feature" for me.

[jorisvandenbossche]

It's rather about too many fields rather than missing fields. But to the point :-), we have some documentation about how read_csv handles malformed files (http://pandas.pydata.org/pandas-docs/stable/io.html#handling-bad-lines + docstring on those keywords), which is: raising an error when there are lines with more fields than the number of columns (or skip the line altogether if you specify a keyword). So IMO it would be nice to explain there that you can also make use of usecols to ignore the excessive data (instead of raising or dropping full line).

BTW, it seems already worked in at least 0.18.1, so it's indeed not a bug 'fix' :-)
But a confirmation of the behaviour in tests (which is good!)

[gfyoung]

@jorisvandenbossche :

1. Ah, indeed csv_reader with limited number of columns should should completely disregard the unused fields #8985 was already a non-issue, but 100% agree that confirmation is good.

2. Fair enough regarding usecols. I'll add an example about that.

[gfyoung]

@jorisvandenbossche : Added example in the docs regarding usecols for handling malformed rows. Ready to merge if there are no other concerns.

[jorisvandenbossche]

@gfyoung Thanks a lot for the addition!