Skip to content

ERR: Raise ValueError when setting scalars in a dataframe with no index ( #16823) #16968

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 8, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/source/whatsnew/v0.21.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -706,6 +706,8 @@ Other API Changes
- Restricted DateOffset keyword arguments. Previously, ``DateOffset`` subclasses allowed arbitrary keyword arguments which could lead to unexpected behavior. Now, only valid arguments will be accepted. (:issue:`17176`).
- Pandas no longer registers matplotlib converters on import. The converters
will be registered and used when the first plot is draw (:issue:`17710`)
- Setting on a column with a scalar value and 0-len index now raises a ``ValueError`` (:issue:`16823`)


.. _whatsnew_0210.deprecations:

Expand Down
12 changes: 8 additions & 4 deletions pandas/core/frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -2531,13 +2531,17 @@ def _ensure_valid_index(self, value):
passed value
"""
# GH5632, make sure that we are a Series convertible
if not len(self.index) and is_list_like(value):
if not len(self.index):
if not is_list_like(value):
# GH16823, Raise an error due to loss of information
raise ValueError('If using all scalar values, you must pass'
' an index')
try:
value = Series(value)
except:
raise ValueError('Cannot set a frame with no defined index '
'and a value that cannot be converted to a '
'Series')
raise ValueError('Cannot set a frame with no defined'
'index and a value that cannot be '
'converted to a Series')

self._data = self._data.reindex_axis(value.index.copy(), axis=1,
fill_value=np.nan)
Expand Down
3 changes: 3 additions & 0 deletions pandas/core/reshape/pivot.py
Original file line number Diff line number Diff line change
Expand Up @@ -454,6 +454,9 @@ def crosstab(index, columns, values=None, rownames=None, colnames=None,

from pandas import DataFrame
df = DataFrame(data, index=common_idx)
if not len(df):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than do this (which is incorrect because the result doesn't have the correct index).

just change next statement to

if len(df) and values is None:
....

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I tried that and got an assertion error because the test was using a new empty dataframe as expected value, and the return value was an empty dataframe with a different index. I'll check again, and if it still breaks I'll change the expected value accordingly, assuming no other tests break.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing the statement to if len(df) and values is None: stills makes the crosstab raise the ValueError I'm implementing in this PR, because in this test len(df) is 0 and values IS None, so when it falls to:

else:
        df['__dummy__'] = values
        kwargs = {'aggfunc': aggfunc}

it's basically doing df['__dummy__'] = None which is what raises the error.

What I think would fix it and preserve the index would be this:

if not len(df):
    return df

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see my comment below.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually just return did here i think is ok (and u had that before)

change and ping on green

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, I'm not sure I understand what you mean. Should I change it back to return an empty DataFrame?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no change back to

if not len(df)
    return df

you need to return the DataFrame with the correct index.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, changed it back :)

return DataFrame(index=common_idx)

if values is None:
df['__dummy__'] = 0
kwargs = {'aggfunc': len, 'fill_value': 0}
Expand Down
5 changes: 5 additions & 0 deletions pandas/tests/frame/test_indexing.py
Original file line number Diff line number Diff line change
Expand Up @@ -721,6 +721,11 @@ def test_setitem_empty_frame_with_boolean(self):
df[df > df2] = 47
assert_frame_equal(df, df2)

def test_setitem_scalars_no_index(self):
# GH16823
df = DataFrame()
pytest.raises(ValueError, df.__setitem__, 'foo', 1)

def test_getitem_empty_frame_with_boolean(self):
# Test for issue #11859

Expand Down
11 changes: 5 additions & 6 deletions pandas/tests/indexing/test_loc.py
Original file line number Diff line number Diff line change
Expand Up @@ -423,15 +423,14 @@ def test_loc_setitem_consistency(self):

def test_loc_setitem_consistency_empty(self):
# empty (essentially noops)
expected = DataFrame(columns=['x', 'y'])
expected['x'] = expected['x'].astype(np.int64)
# GH16823
df = DataFrame(columns=['x', 'y'])
df.loc[:, 'x'] = 1
tm.assert_frame_equal(df, expected)
with tm.assert_raises_regex(ValueError, 'If using all scalar values'):
df.loc[:, 'x'] = 1

df = DataFrame(columns=['x', 'y'])
df['x'] = 1
tm.assert_frame_equal(df, expected)
with tm.assert_raises_regex(ValueError, 'If using all scalar values'):
df['x'] = 1

def test_loc_setitem_consistency_slice_column_len(self):
# .loc[:,column] setting with slice == len of the column
Expand Down
20 changes: 6 additions & 14 deletions pandas/tests/indexing/test_partial.py
Original file line number Diff line number Diff line change
Expand Up @@ -575,24 +575,16 @@ def f():
def test_partial_set_empty_frame_row(self):
# GH5720, GH5744
# don't create rows when empty
expected = DataFrame(columns=['A', 'B', 'New'],
index=pd.Index([], dtype='int64'))
expected['A'] = expected['A'].astype('int64')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok this is fine here.

expected['B'] = expected['B'].astype('float64')
expected['New'] = expected['New'].astype('float64')

df = DataFrame({"A": [1, 2, 3], "B": [1.2, 4.2, 5.2]})
y = df[df.A > 5]
y['New'] = np.nan
tm.assert_frame_equal(y, expected)
# tm.assert_frame_equal(y,expected)
# GH16823
# Setting a column with a scalar and no index should raise
with tm.assert_raises_regex(ValueError, 'If using all scalar values'):
y['New'] = np.nan

expected = DataFrame(columns=['a', 'b', 'c c', 'd'])
expected['d'] = expected['d'].astype('int64')
df = DataFrame(columns=['a', 'b', 'c c'])
df['d'] = 3
tm.assert_frame_equal(df, expected)
tm.assert_series_equal(df['c c'], Series(name='c c', dtype=object))
with tm.assert_raises_regex(ValueError, 'If using all scalar values'):
df['d'] = 3

# reindex columns is ok
df = DataFrame({"A": [1, 2, 3], "B": [1.2, 4.2, 5.2]})
Expand Down
3 changes: 2 additions & 1 deletion pandas/tests/reshape/test_pivot.py
Original file line number Diff line number Diff line change
Expand Up @@ -1226,7 +1226,8 @@ def test_crosstab_no_overlap(self):
s2 = pd.Series([4, 5, 6], index=[4, 5, 6])

actual = crosstab(s1, s2)
expected = pd.DataFrame()
expected = pd.DataFrame(
index=pd.Index([], dtype='int64')).astype('int64')

tm.assert_frame_equal(actual, expected)

Expand Down