#26545 Fix: same .tsv file, get different data-frame structure using engine 'python' and 'c'

[luckydenis]

• closes same .tsv file, get different data-frame structure using engine 'python' and 'c' #26545
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

BugFix:
When using engine='python', columns were handled incorrectly if the first header had in the bom.

Bug:

In [1]: import pandas as pd                                                             

In [2]: pd.read_csv('test.txt', engine='python', delimiter='\t')              
Out[2]: 
Empty DataFrame
Columns: [Project ID]
Index: []

In [3]: pd.read_csv('test.txt', engine='python', delimiter='\t').shape        
Out[3]: (0, 1)

In [4]: pd.read_csv('test.txt', delimiter='\t')                               
Out[4]: 
Empty DataFrame
Columns: [Project ID, Project Name, Product Name]
Index: []

In [5]: pd.read_csv('test, delimiter='\t').shape                         
Out[5]: (0, 3)

Fix:

In [1]: import pandas as pd                                                             

In [2]: pd.read_csv('test.txt', engine='python', delimiter='\t')              
Out[2]: 
Empty DataFrame
Columns: [Project ID, Project Name, Product Name]
Index: []

In [3]: pd.read_csv('test.txt', engine='python', delimiter='\t').shape        
Out[3]: (0, 3)

In [4]: pd.read_csv('test.txt', delimiter='\t')                               
Out[4]: 
Empty DataFrame
Columns: [Project ID, Project Name, Product Name]
Index: []

In [5]: pd.read_csv('test, delimiter='\t').shape                         
Out[5]: (0, 3)

[pep8speaks]

Hello @luckydenis! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-06-11 07:36:17 UTC

[WillAyd]

This closes an issue right? If so can you update the OP to reflect that

[codecov]

Codecov Report

Merging #26634 into master will decrease coverage by 50.09%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26634      +/-   ##
==========================================
- Coverage   91.88%   41.78%   -50.1%     
==========================================
  Files         174      174              
  Lines       50692    50692              
==========================================
- Hits        46576    21182   -25394     
- Misses       4116    29510   +25394

Flag	Coverage Δ
#multiple	?
#single	41.78% <ø> (-0.11%)	⬇️

Impacted Files	Coverage Δ
pandas/io/formats/latex.py	0% <0%> (-100%)	⬇️
pandas/io/sas/sas_constants.py	0% <0%> (-100%)	⬇️
pandas/core/groupby/categorical.py	0% <0%> (-100%)	⬇️
pandas/tseries/plotting.py	0% <0%> (-100%)	⬇️
pandas/tseries/converter.py	0% <0%> (-100%)	⬇️
pandas/io/formats/html.py	0% <0%> (-99.37%)	⬇️
pandas/io/sas/sas7bdat.py	0% <0%> (-91.16%)	⬇️
pandas/io/sas/sas_xport.py	0% <0%> (-90.1%)	⬇️
pandas/core/sparse/scipy_sparse.py	10.14% <0%> (-89.86%)	⬇️
pandas/core/tools/numeric.py	10.14% <0%> (-89.86%)	⬇️
... and 128 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d124ea...0816572. Read the comment docs.

[codecov]

Codecov Report

Merging #26634 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26634      +/-   ##
==========================================
- Coverage   91.72%   91.71%   -0.01%     
==========================================
  Files         178      178              
  Lines       50779    50779              
==========================================
- Hits        46578    46574       -4     
- Misses       4201     4205       +4

Flag	Coverage Δ
#multiple	90.31% <ø> (ø)	⬆️
#single	41.19% <ø> (-0.09%)	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	78.94% <0%> (-10.53%)	⬇️
pandas/core/frame.py	96.88% <0%> (-0.12%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 157a4e3...fb010d5. Read the comment docs.

[luckydenis]

@WillAyd, I made a correction on your comments. Look please)

[WillAyd]

Thanks for the updates - can you also add a whatsnew note for 0.25?

cc @gfyoung if you care to take a look

[WillAyd]

@luckydenis could you also check what test_utf8_bom is doing in the test module? It looks to cover the same intention as test added so want to make sure we understand the difference and clarify accordingly

[luckydenis]

In this context, I'm not sure that they are different, but my PR cures this error, which tm method should be used to check it? The error was in the method that was cleaning from the bom.

In [1]: import pandas as pd                                                             

In [2]: pd.read_csv('test.txt', engine='python', delimiter='\t')              
Out[2]: 
Empty DataFrame
Columns: [Project ID]
Index: []

In [3]: pd.read_csv('test.txt', engine='python', delimiter='\t').shape        
Out[3]: (0, 1)

In [4]: pd.read_csv('test.txt', delimiter='\t')                               
Out[4]: 
Empty DataFrame
Columns: [Project ID, Project Name, Product Name]
Index: []

In [5]: pd.read_csv('test, delimiter='\t').shape                         
Out[5]: (0, 3)

[WillAyd]

I think this is good if you can add a whatsnew for 0.25 then lgtm!

[jreback]

@gfyoung ok with this?

[gfyoung]

@jreback : Looks fine to me, just need the whatsnew as @WillAyd said

[luckydenis]

Yeah, I'll add a description in the whatsnew, just haven't had time.

[luckydenis]

@jreback, @WillAyd, @gfyoung I added, look please.

[WillAyd]

lgtm @gfyoung

[gfyoung]

@jreback if we have anything else

[jreback]

lgtm. @luckydenis if you can merge master and ping on green to resolve the conflict.

[jreback]

thanks @luckydenis