BUG: DataFrame.at must fail when adding more than one value

[johannes-mueller]

Greetings from EuroScipy 2022 and thanks to @jorisvandenbossche for support.

The problem occurred because when adding a new column to a DataFrame using
DataFrame.at, DataFrame._set_value falls back to DataFrame.loc before the
InvalidIndexError is raised.

The proposed solution makes sure that both, the column index and the row index
are determined so that both have a chance to raise.

• closes BUG: using at indexer reassign existing field error #48224
• Added an entry in doc/source/whatsnew/v1.6.0.rst file if fixing a bug or adding a new feature.
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.

[phofl]

lgtm in general. Could you check if this impacts #48323?

[jorisvandenbossche]

if we consider this as a not-that-notable bug fix (although it did work "correctly" in the past, it was just not failing and doing too much for what we want at to do)

On a second thought, I was just checking if this was also working in the past, and it seems the test example here has "correctly" worked basically forever (I tested down to 0.18). So we should maybe consider deprecating this first?

[johannes-mueller]

if we consider this as a not-that-notable bug fix (although it did work "correctly" in the past, it was just not failing and doing too much for what we want at to do)

On a second thought, I was just checking if this was also working in the past, and it seems the test example here has "correctly" worked basically forever (I tested down to 0.18). So we should maybe consider deprecating this first?

Since 0.23 there is the sentence Use at if you only need to get or set a single value in a DataFrame or Series. in the docs. So we could consider it deprecated since then.

But I wouldn't mind deprecating "officially".

[phofl]

I would disallow immediately. Using the same indexer again raises, so we would fix an inconsistency on top of that.

[johannes-mueller]

lgtm in general. Could you check if this impacts #48323?

If we really decide that .at should never add new columns or rows to a DataFrame we probably should not merge this. In that case we must fix the user guide, though as @jorisvandenbossche showed, that it allows the enlargement by .at.

If we decide to merge this, I would suggest also fixing the docstring of .at. AFAICS it now never raises a KeyError because these are all caught in ._set_value which then falls back to .loc.

So decision needed.

[srotondo]

@johannes-mueller In pandas/tests/indexing/test_at.py, there is a test named test_at_setitem_multiindex. During this test, the line df.at[0, "a"] = 10 successfully modifies two columns at once. I believe your code change here should prevent this from succeeding, right? I can tell from the code checks here and modified files that this test still succeeds and that you didn't change this test at all. Also, changing this line to x = df.at[0, "a"] and trying to only get this value will raise an error, so I believe causing this setting action to fail should be the correct course of action.

[srotondo]

If we really decide that .at should never add new columns or rows to a DataFrame we probably should not merge this. In that case we must fix the user guide, though as @jorisvandenbossche showed, that it allows the enlargement by .at.

I am working on a PR that will do this exact thing, including modifying the user guide and documentation. I'll link it here once I'm done with it so we can decide what to do to fix this bug.

[srotondo]

I've finally submitted my PR. You can find it at #48542.

[johannes-mueller]

Superseded by #48542