Skip to content

BUG: Fix csv_read bugs when using empty input. GH10467 & GH10413 #10469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 7, 2015
Merged

BUG: Fix csv_read bugs when using empty input. GH10467 & GH10413 #10469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0eb1587
BUG: Fix csv_read bugs when using empty input. GH10467 & GH10413
Jun 29, 2015
4b2ceeb
BUG: Fix csv_read bug with reversed index_col list.
santegoeds Jul 2, 2015
8523105
Add addition test for pandas.csv_read to test variations of index_col…
santegoeds Jul 6, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/source/whatsnew/v0.17.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,3 +134,5 @@ Bug Fixes

- Bug in `pandas.concat` with ``axis=0`` when column is of dtype ``category`` (:issue:`10177`)
- Bug in ``read_msgpack`` where input type is not always checked (:issue:`10369`)

- Bug in `pandas.read_csv` with ``index_col=False`` or with ``index_col=['a', 'b']`` (:issue:`10413`, :issue:`10467`)
13 changes: 7 additions & 6 deletions pandas/io/parsers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2223,13 +2223,14 @@ def _clean_index_names(columns, index_col):
def _get_empty_meta(columns, index_col, index_names):
columns = list(columns)

if index_col is not None:
index = MultiIndex.from_arrays([[]] * len(index_col),
names=index_names)
for n in index_col:
columns.pop(n)
else:
if index_col is None or index_col is False:
index = Index([])
else:
index_col = list(index_col)
index = MultiIndex.from_arrays([[]] * len(index_col), names=index_names)
index_col.sort()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does this need sorting? why not just pop them in order?

e.g.
columns = columns - Index(index_col)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It needs sorting so that the calculated column index, i.e. n-i in the pop, evaluates to the correct value even for a multiindex that's specified with decrementing index columns.

E.g. Without the sort and for index_col=[1,0], the loop would try to pop first column 1 - 0 = 0 and then 0 - 1 = -1.

I expect I'm missing something but columns = columns - Index(index_col) raises an error for me (*** TypeError: can only perform ops with scalar values)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry old syntax

try

columns.difference(Index(index_col))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

columns is a list of strings and index_col is a list of integers. I don't think this will work either.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think need some more tests then
index_col can also be a list of strings

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this PR we already have tests that specify a multiindex with either a list of strings or a list of ints (see test_empty_with_multiindex and test_empty_with_reversed_multiindex).

I haven't completely stepped through the code but afaics, function _clean_index_names is called prior to get_empty_meta and converts index_col as a list of strings into a list of ints.

for i, n in enumerate(index_col):
columns.pop(n-i)

return index, columns, {}

Expand Down
75 changes: 75 additions & 0 deletions pandas/io/tests/test_parsers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2301,6 +2301,81 @@ def test_empty_with_index(self):
expected = DataFrame([], columns=['y'], index=Index([], name='x'))
tm.assert_frame_equal(result, expected)

def test_emtpy_with_multiindex(self):
# GH 10467
data = 'x,y,z'
result = self.read_csv(StringIO(data), index_col=['x', 'y'])
expected = DataFrame([], columns=['z'],
index=MultiIndex.from_arrays([[]] * 2, names=['x', 'y']))
tm.assert_frame_equal(result, expected)

def test_empty_with_reversed_multiindex(self):
data = 'x,y,z'
result = self.read_csv(StringIO(data), index_col=[1, 0])
expected = DataFrame([], columns=['z'],
index=MultiIndex.from_arrays([[]] * 2, names=['y', 'x']))
tm.assert_frame_equal(result, expected)

def test_empty_index_col_scenarios(self):
data = 'x,y,z'

# None, no index
index_col, expected = None, DataFrame([], columns=list('xyz')),
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

# False, no index
index_col, expected = False, DataFrame([], columns=list('xyz')),
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

# int, first column
index_col, expected = 0, DataFrame([], columns=['y', 'z'], index=Index([], name='x'))
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

# int, not first column
index_col, expected = 1, DataFrame([], columns=['x', 'z'], index=Index([], name='y'))
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

# str, first column
index_col, expected = 'x', DataFrame([], columns=['y', 'z'], index=Index([], name='x'))
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

# str, not the first column
index_col, expected = 'y', DataFrame([], columns=['x', 'z'], index=Index([], name='y'))
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

# list of int
index_col, expected = [0, 1], DataFrame([], columns=['z'],
index=MultiIndex.from_arrays([[]] * 2, names=['x', 'y']))
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

# list of str
index_col, expected = (
['x', 'y'],
DataFrame([], columns=['z'], index=MultiIndex.from_arrays([[]] * 2, names=['x', 'y']))
)
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

# list of int, reversed sequence
index_col, expected = (
[1, 0],
DataFrame([], columns=['z'], index=MultiIndex.from_arrays([[]] * 2, names=['y', 'x']))
)
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

# list of str, reversed sequence
index_col, expected = (
['y', 'x'],
DataFrame([], columns=['z'], index=MultiIndex.from_arrays([[]] * 2, names=['y', 'x']))
)
tm.assert_frame_equal(self.read_csv(StringIO(data), index_col=index_col), expected)

def test_empty_with_index_col_false(self):
# GH 10413
data = 'x,y'
result = self.read_csv(StringIO(data), index_col=False)
expected = DataFrame([], columns=['x', 'y'])
tm.assert_frame_equal(result, expected)

def test_float_parser(self):
# GH 9565
data = '45e-1,4.5,45.,inf,-inf'
Expand Down