Skip to content

BUG: Fixed strange behaviour of pd.DataFrame.drop() with inplace argu… #30501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Mar 26, 2020
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
55ababc
BUG: Fixed strange behaviour of pd.DataFrame.drop() with inplace argu…
Dec 27, 2019
1dc7e72
Revert "DOC: standardize wording for changed default args (#30493)"
Dec 27, 2019
3c48c5d
Merge branch 'master' of https://github.com/pandas-dev/pandas
Dec 28, 2019
665fa27
moved new test to test_operators.py and added new bool argument to ND…
Dec 29, 2019
cc5e480
fixed black formatting
Dec 29, 2019
35e3e0f
Merge branch 'master' into master
rjfs Jan 27, 2020
d2a8b90
moved and refactored new test; implemented new solution by resetting …
Jan 27, 2020
72d3815
Merge branch 'master' of https://github.com/rjfs/pandas
Jan 27, 2020
f0dbe12
removed some old changes
Jan 27, 2020
00a5c7f
removed some old changes
Jan 27, 2020
f33f55f
removed some old changes
Jan 27, 2020
27cbd51
Update v1.0.0.rst
rjfs Jan 27, 2020
ddf890f
Update v1.0.0.rst
rjfs Jan 27, 2020
a5182f4
Merge remote-tracking branch 'upstream/master'
Feb 10, 2020
a199bdf
Merge branch 'master' of https://github.com/rjfs/pandas
Feb 10, 2020
987c274
Merge remote-tracking branch 'upstream/master'
Mar 7, 2020
0b979f9
restructured test
Mar 7, 2020
5b59b01
Merge remote-tracking branch 'upstream/master'
Mar 14, 2020
53cd24a
Update v1.1.0.rst
rjfs Mar 14, 2020
a7ce080
Merge remote-tracking branch 'upstream/master'
Mar 14, 2020
54bf552
Merge remote-tracking branch 'upstream/master'
Mar 14, 2020
aed33ba
Merge branch 'master' into master
rjfs Mar 14, 2020
7a558ad
Merge branch 'master' into master
jreback Mar 25, 2020
7c08e9b
Merge branch 'master' into master
rjfs Mar 26, 2020
76131c0
merged with master
Mar 26, 2020
a220dc4
Merge remote-tracking branch 'upstream/master'
Mar 26, 2020
5606519
merge with master
Mar 26, 2020
9bdf454
merge with master
Mar 26, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions doc/source/whatsnew/v1.0.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -567,13 +567,13 @@ or ``matplotlib.Axes.plot``. See :ref:`plotting.formatters` for more.
- :meth:`DataFrame.hist` and :meth:`Series.hist` no longer allows ``figsize="default"``, specify figure size by passinig a tuple instead (:issue:`30003`)
- Floordiv of integer-dtyped array by :class:`Timedelta` now raises ``TypeError`` (:issue:`21036`)
- :class:`TimedeltaIndex` and :class:`DatetimeIndex` no longer accept non-nanosecond dtype strings like "timedelta64" or "datetime64", use "timedelta64[ns]" and "datetime64[ns]" instead (:issue:`24806`)
- Changed the default "skipna" argument in :func:`pandas.api.types.infer_dtype` from ``False`` to ``True`` (:issue:`24050`)
- :func:`pandas.api.types.infer_dtype` argument ``skipna`` defaults to ``True`` instead of ``False`` (:issue:`24050`)
- Removed :attr:`Series.ix` and :attr:`DataFrame.ix` (:issue:`26438`)
- Removed :meth:`Index.summary` (:issue:`18217`)
- Removed the previously deprecated keyword "fastpath" from the :class:`Index` constructor (:issue:`23110`)
- Removed :meth:`Series.get_value`, :meth:`Series.set_value`, :meth:`DataFrame.get_value`, :meth:`DataFrame.set_value` (:issue:`17739`)
- Removed :meth:`Series.compound` and :meth:`DataFrame.compound` (:issue:`26405`)
- Changed the default "inplace" argument in :meth:`DataFrame.set_index` and :meth:`Series.set_axis` from ``None`` to ``False`` (:issue:`27600`)
- Changed the the default value of `inplace` in :meth:`DataFrame.set_index` and :meth:`Series.set_axis`. It now defaults to ``False`` (:issue:`27600`)
- Removed :attr:`Series.cat.categorical`, :attr:`Series.cat.index`, :attr:`Series.cat.name` (:issue:`24751`)
- Removed the previously deprecated keyword "box" from :func:`to_datetime` and :func:`to_timedelta`; in addition these now always returns :class:`DatetimeIndex`, :class:`TimedeltaIndex`, :class:`Index`, :class:`Series`, or :class:`DataFrame` (:issue:`24486`)
- :func:`to_timedelta`, :class:`Timedelta`, and :class:`TimedeltaIndex` no longer allow "M", "y", or "Y" for the "unit" argument (:issue:`23264`)
Expand Down Expand Up @@ -613,7 +613,7 @@ or ``matplotlib.Axes.plot``. See :ref:`plotting.formatters` for more.
- :func:`read_excel` no longer allows an integer value for the parameter ``usecols``, instead pass a list of integers from 0 to ``usecols`` inclusive (:issue:`23635`)
- Removed the previously deprecated keyword "convert_datetime64" from :meth:`DataFrame.to_records` (:issue:`18902`)
- Removed :meth:`IntervalIndex.from_intervals` in favor of the :class:`IntervalIndex` constructor (:issue:`19263`)
- Changed the default "keep_tz" argument in :meth:`DatetimeIndex.to_series` from ``None`` to ``True`` (:issue:`23739`)
- Changed the default value for the "keep_tz" argument in :meth:`DatetimeIndex.to_series` to ``True`` (:issue:`23739`)
- Removed :func:`api.types.is_period` and :func:`api.types.is_datetimetz` (:issue:`23917`)
- Ability to read pickles containing :class:`Categorical` instances created with pre-0.16 version of pandas has been removed (:issue:`27538`)
- Removed :func:`pandas.tseries.plotting.tsplot` (:issue:`18627`)
Expand All @@ -622,7 +622,7 @@ or ``matplotlib.Axes.plot``. See :ref:`plotting.formatters` for more.
- Removed the previously deprecated ``FrozenNDArray`` class in ``pandas.core.indexes.frozen`` (:issue:`29335`)
- Removed the previously deprecated keyword "nthreads" from :func:`read_feather`, use "use_threads" instead (:issue:`23053`)
- Removed :meth:`Index.is_lexsorted_for_tuple` (:issue:`29305`)
- Removed support for nested renaming in :meth:`DataFrame.aggregate`, :meth:`Series.aggregate`, :meth:`DataFrameGroupBy.aggregate`, :meth:`SeriesGroupBy.aggregate`, :meth:`Rolling.aggregate` (:issue:`29608`)
- Removed support for nexted renaming in :meth:`DataFrame.aggregate`, :meth:`Series.aggregate`, :meth:`DataFrameGroupBy.aggregate`, :meth:`SeriesGroupBy.aggregate`, :meth:`Rolling.aggregate` (:issue:`29608`)
- Removed :meth:`Series.valid`; use :meth:`Series.dropna` instead (:issue:`18800`)
- Removed :attr:`DataFrame.is_copy`, :attr:`Series.is_copy` (:issue:`18812`)
- Removed :meth:`DataFrame.get_ftype_counts`, :meth:`Series.get_ftype_counts` (:issue:`18243`)
Expand All @@ -634,7 +634,7 @@ or ``matplotlib.Axes.plot``. See :ref:`plotting.formatters` for more.
- Removed :meth:`DatetimeIndex.asobject`, :meth:`TimedeltaIndex.asobject`, :meth:`PeriodIndex.asobject`, use ``astype(object)`` instead (:issue:`29801`)
- Removed the previously deprecated keyword "order" from :func:`factorize` (:issue:`19751`)
- Removed the previously deprecated keyword "encoding" from :func:`read_stata` and :meth:`DataFrame.to_stata` (:issue:`21400`)
- Changed the default "sort" argument in :func:`concat` from ``None`` to ``False`` (:issue:`20613`)
- In :func:`concat` the default value for ``sort`` has been changed from ``None`` to ``False`` (:issue:`20613`)
- Removed the previously deprecated keyword "raise_conflict" from :meth:`DataFrame.update`, use "errors" instead (:issue:`23585`)
- Removed the previously deprecated keyword "n" from :meth:`DatetimeIndex.shift`, :meth:`TimedeltaIndex.shift`, :meth:`PeriodIndex.shift`, use "periods" instead (:issue:`22458`)
- Removed the previously deprecated keywords "how", "fill_method", and "limit" from :meth:`DataFrame.resample` (:issue:`30139`)
Expand All @@ -650,15 +650,16 @@ or ``matplotlib.Axes.plot``. See :ref:`plotting.formatters` for more.
- :meth:`Categorical.ravel` returns a :class:`Categorical` instead of a ``ndarray`` (:issue:`27199`)
- The 'outer' method on Numpy ufuncs, e.g. ``np.subtract.outer`` operating on :class:`Series` objects is no longer supported, and will raise ``NotImplementedError`` (:issue:`27198`)
- Removed :meth:`Series.get_dtype_counts` and :meth:`DataFrame.get_dtype_counts` (:issue:`27145`)
- Changed the default "fill_value" argument in :meth:`Categorical.take` from ``True`` to ``False`` (:issue:`20841`)
- Changed the default value for the `raw` argument in :func:`Series.rolling().apply() <pandas.core.window.Rolling.apply>`, :func:`DataFrame.rolling().apply() <pandas.core.window.Rolling.apply>`, :func:`Series.expanding().apply() <pandas.core.window.Expanding.apply>`, and :func:`DataFrame.expanding().apply() <pandas.core.window.Expanding.apply>` from ``None`` to ``False`` (:issue:`20584`)
- Changed the default ``fill_value`` in :meth:`Categorical.take` from ``True`` to ``False`` (:issue:`20841`)
- Changed the default value for the `raw` argument in :func:`Series.rolling().apply() <pandas.core.window.Rolling.apply>`, :func:`DataFrame.rolling().apply() <pandas.core.window.Rolling.apply>`,
- :func:`Series.expanding().apply() <pandas.core.window.Expanding.apply>`, and :func:`DataFrame.expanding().apply() <pandas.core.window.Expanding.apply>` to ``False`` (:issue:`20584`)
- Removed deprecated behavior of :meth:`Series.argmin` and :meth:`Series.argmax`, use :meth:`Series.idxmin` and :meth:`Series.idxmax` for the old behavior (:issue:`16955`)
- Passing a tz-aware ``datetime.datetime`` or :class:`Timestamp` into the :class:`Timestamp` constructor with the ``tz`` argument now raises a ``ValueError`` (:issue:`23621`)
- Removed :attr:`Series.base`, :attr:`Index.base`, :attr:`Categorical.base`, :attr:`Series.flags`, :attr:`Index.flags`, :attr:`PeriodArray.flags`, :attr:`Series.strides`, :attr:`Index.strides`, :attr:`Series.itemsize`, :attr:`Index.itemsize`, :attr:`Series.data`, :attr:`Index.data` (:issue:`20721`)
- Changed :meth:`Timedelta.resolution` to match the behavior of the standard library ``datetime.timedelta.resolution``, for the old behavior, use :meth:`Timedelta.resolution_string` (:issue:`26839`)
- Removed :attr:`Timestamp.weekday_name`, :attr:`DatetimeIndex.weekday_name`, and :attr:`Series.dt.weekday_name` (:issue:`18164`)
- Removed the previously deprecated keyword "errors" in :meth:`Timestamp.tz_localize`, :meth:`DatetimeIndex.tz_localize`, and :meth:`Series.tz_localize` (:issue:`22644`)
- Changed the default "ordered" argument in :class:`CategoricalDtype` from ``None`` to ``False`` (:issue:`26336`)
- Changed the default value for ``ordered`` in :class:`CategoricalDtype` from ``None`` to ``False`` (:issue:`26336`)
- :meth:`Series.set_axis` and :meth:`DataFrame.set_axis` now require "labels" as the first argument and "axis" as an optional named parameter (:issue:`30089`)
- Removed :func:`to_msgpack`, :func:`read_msgpack`, :meth:`DataFrame.to_msgpack`, :meth:`Series.to_msgpack` (:issue:`27103`)
- Removed :meth:`Series.compress` (:issue:`21930`)
Expand Down
16 changes: 11 additions & 5 deletions pandas/core/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -3215,7 +3215,10 @@ def _get_cacher(self):
return cacher

def _maybe_update_cacher(
self, clear: bool_t = False, verify_is_copy: bool_t = True
self,
clear: bool_t = False,
verify_is_copy: bool_t = True,
change_cache: bool_t = True,
) -> None:
"""
See if we need to update our parent cacher if clear, then clear our
Expand All @@ -3227,8 +3230,9 @@ def _maybe_update_cacher(
Clear the item cache.
verify_is_copy : bool, default True
Provide is_copy checks.
change_cache: bool, default True
If True, then cache may be changed
"""

cacher = getattr(self, "_cacher", None)
if cacher is not None:
ref = cacher[1]()
Expand All @@ -3237,7 +3241,7 @@ def _maybe_update_cacher(
# a copy
if ref is None:
del self._cacher
else:
elif change_cache:
# Note: we need to call ref._maybe_cache_changed even in the
# case where it will raise. (Uh, not clear why)
try:
Expand Down Expand Up @@ -3931,22 +3935,24 @@ def _drop_axis(self, labels, axis, level=None, errors: str = "raise"):

return result

def _update_inplace(self, result, verify_is_copy: bool_t = True) -> None:
def _update_inplace(self, result, verify_is_copy: bool_t = True, **kwargs) -> None:
"""
Replace self internals with result.

Parameters
----------
verify_is_copy : bool, default True
Provide is_copy checks.
**kwargs
Passed to self._maybe_update_cacher
"""
# NOTE: This does *not* call __finalize__ and that's an explicit
# decision that we may revisit in the future.

self._reset_cache()
self._clear_item_cache()
self._data = getattr(result, "_data", result)
self._maybe_update_cacher(verify_is_copy=verify_is_copy)
self._maybe_update_cacher(verify_is_copy=verify_is_copy, **kwargs)

def add_prefix(self, prefix: str):
"""
Expand Down
4 changes: 3 additions & 1 deletion pandas/core/ops/methods.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,9 @@ def f(self, other):
# this makes sure that we are aligned like the input
# we are updating inplace so we want to ignore is_copy
self._update_inplace(
result.reindex_like(self, copy=False)._data, verify_is_copy=False
result.reindex_like(self, copy=False)._data,
verify_is_copy=False,
change_cache=False,
)

return self
Expand Down
22 changes: 22 additions & 0 deletions pandas/tests/frame/test_operators.py
Original file line number Diff line number Diff line change
Expand Up @@ -829,6 +829,28 @@ def test_inplace_ops_identity2(self, op):
expected = id(df)
assert id(df) == expected

def test_inplace_drop_and_add(self):
# GH 30484
# Get expected df
expected = pd.DataFrame({})
expected["x1"] = [1, 2, 3, 4, 5]
expected["x2"] = [0, 0, 0, 1, 1]
expected["target"] = [10, 20, 30, 40, 50]
y_expected = expected["target"]
expected = expected.drop("target", axis=1, inplace=False)
y_expected += np.min(y_expected)
# Get tested df
df = pd.DataFrame({})
df["x1"] = [1, 2, 3, 4, 5]
df["x2"] = [0, 0, 0, 1, 1]
df["target"] = [10, 20, 30, 40, 50]
y = df["target"]
df.drop("target", axis=1, inplace=True)
y += np.min(y)
# compare
tm.assert_frame_equal(df, expected)
tm.assert_series_equal(y, y_expected)

def test_alignment_non_pandas(self):
index = ["A", "B", "C"]
columns = ["X", "Y", "Z"]
Expand Down