BUG: Python Parser skipping over items if BOM present in first element of header #36365
Conversation
pandas/io/parsers.py
Outdated
| @@ -2876,7 +2876,7 @@ def _check_for_bom(self, first_row): | |||
| return [new_row] + first_row[1:] | |||
|
|
|||
| elif len(first_row_bom) > 1: | |||
| return [first_row_bom[1:]] | |||
| return [first_row_bom[1:]] + first_row[1:] | |||
There was a problem hiding this comment.
Hmm I don't find this very clear why we would do this - can you try refactoring the code above this to better suit the requirement?
There was a problem hiding this comment.
from what I understand
the code block above removes the BOM from the first element of the row and removes the quotes if it was a quoted string. i.e. if it was of the format <BOM><QUOTE>abc<QUOTE>def then it extracts out abcdef
I've tried to refactor it slightly
| @@ -2127,6 +2127,12 @@ def test_first_row_bom(all_parsers): | |||
| expected = DataFrame(columns=["Head1", "Head2", "Head3"]) | |||
| tm.assert_frame_equal(result, expected) | |||
|
|
|||
| # see gh-36343 | |||
There was a problem hiding this comment.
can you make this a new test
There was a problem hiding this comment.
done, I think test failures are unrelated. https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=42772&view=ms.vss-test-web.build-test-results-tab&runId=2173848&resultId=150762&paneView=debug
|
@gfyoung if you'd have a look |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
thanks @asishm |
|
k thanks @asishm yeah there might be some others but IIRC those are for the c-parser |
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diff