BUG: read_csv raising IndexError with multiple header cols, specified index_col and no data rows #38325
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… index_col and no data rows
jbrockmendel
reviewed
Dec 7, 2020
doc/source/whatsnew/v1.2.0.rst
Outdated
| @@ -729,6 +729,7 @@ I/O | |||
| - :meth:`read_fwf` was inferring compression with ``compression=None`` which was not consistent with the other :meth:``read_*`` functions (:issue:`37909`) | |||
There was a problem hiding this comment.
i think italics are messed up bc the "read*" should have single-ticks instead of double-ticks
jbrockmendel
reviewed
Dec 7, 2020
doc/source/whatsnew/v1.2.0.rst
Outdated
| @@ -729,6 +729,7 @@ I/O | |||
| - :meth:`read_fwf` was inferring compression with ``compression=None`` which was not consistent with the other :meth:``read_*`` functions (:issue:`37909`) | |||
| - :meth:`DataFrame.to_html` was ignoring ``formatters`` argument for ``ExtensionDtype`` columns (:issue:`36525`) | |||
| - Bumped minimum xarray version to 0.12.3 to avoid reference to the removed ``Panel`` class (:issue:`27101`) | |||
| - Bug in :meth:`read_csv` raising ``IndexError`` with multiple header columns and ``index_col`` specified when file has no data rows (:issue:`38292`) | |||
There was a problem hiding this comment.
Thx, should we change the other too then?
There was a problem hiding this comment.
Unforunately was a bit slow :(
jreback
requested changes
Dec 8, 2020
doc/source/whatsnew/v1.2.0.rst
Outdated
| @@ -726,9 +726,10 @@ I/O | |||
| - :meth:`DataFrame.to_excel`, :meth:`Series.to_excel`, :meth:`DataFrame.to_markdown`, and :meth:`Series.to_markdown` now support writing to fsspec URLs such as S3 and Google Cloud Storage (:issue:`33987`) | |||
| - Bug in :func:`read_fwf` with ``skip_blank_lines=True`` was not skipping blank lines (:issue:`37758`) | |||
| - Parse missing values using :func:`read_json` with ``dtype=False`` to ``NaN`` instead of ``None`` (:issue:`28501`) | |||
| - :meth:`read_fwf` was inferring compression with ``compression=None`` which was not consistent with the other :meth:``read_*`` functions (:issue:`37909`) | |||
| - :meth:`read_fwf` was inferring compression with ``compression=None`` which was not consistent with the other :meth:`read_*` functions (:issue:`37909`) | |||
There was a problem hiding this comment.
will need to move to 1.3 (avail shortly)
There was a problem hiding this comment.
can you remove from 1.2 :->
pandas/io/parsers.py
Outdated
| # list of Nones GH#38292 | ||
| index_names = [None] * len(ic) | ||
| else: | ||
| index_names, _, _ = _clean_index_names( |
There was a problem hiding this comment.
can you generate this in _clean_index_names instead?
| @@ -222,3 +222,18 @@ def test_index_col_large_csv(all_parsers): | |||
| result = parser.read_csv(path, index_col=[0]) | |||
|
|
|||
| tm.assert_frame_equal(result, df.set_index("a")) | |||
|
|
|||
|
|
|||
| def test_index_col_multiindex_columns_no_data(all_parsers): | |||
There was a problem hiding this comment.
can you add all the cases form the op (3 working & the non-working)
jreback
requested changes
Dec 10, 2020
doc/source/whatsnew/v1.2.0.rst
Outdated
| @@ -726,9 +726,10 @@ I/O | |||
| - :meth:`DataFrame.to_excel`, :meth:`Series.to_excel`, :meth:`DataFrame.to_markdown`, and :meth:`Series.to_markdown` now support writing to fsspec URLs such as S3 and Google Cloud Storage (:issue:`33987`) | |||
| - Bug in :func:`read_fwf` with ``skip_blank_lines=True`` was not skipping blank lines (:issue:`37758`) | |||
| - Parse missing values using :func:`read_json` with ``dtype=False`` to ``NaN`` instead of ``None`` (:issue:`28501`) | |||
| - :meth:`read_fwf` was inferring compression with ``compression=None`` which was not consistent with the other :meth:``read_*`` functions (:issue:`37909`) | |||
| - :meth:`read_fwf` was inferring compression with ``compression=None`` which was not consistent with the other :meth:`read_*` functions (:issue:`37909`) | |||
There was a problem hiding this comment.
can you remove from 1.2 :->
|
@jreback green |
jreback
approved these changes
Dec 12, 2020
|
lgtm, resolved conflict, though there is another PR that might conflict on whatsnew, but will see. |
|
Yeah, thanks for merging. As soon as we got a few messages in there we can vary the order a bit to avoid this |
|
thanks |
luckyvs1
pushed a commit
to luckyvs1/pandas
that referenced
this pull request
Jan 20, 2021
… index_col and no data rows (pandas-dev#38325) * BUG: read_csv raising IndexError with multiple header cols, specified index_col and no data rows * Adress comments * Add tests and move whatsnew * Revert change Co-authored-by: Jeff Reback <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diffParser returned an empty list instead of a list filled with None, which caused the IndexError.