Skip to content

BUG: EA inplace add (and other ops) with non-EA arg broken #37910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
marco-neumann-by opened this issue Nov 17, 2020 · 2 comments · Fixed by #37986
Closed
3 tasks done

BUG: EA inplace add (and other ops) with non-EA arg broken #37910

marco-neumann-by opened this issue Nov 17, 2020 · 2 comments · Fixed by #37986
Labels
ExtensionArray Extending pandas with custom dtypes or arrays. Regression Functionality that used to work in a prior pandas version
Milestone

Comments

@marco-neumann-by
Copy link

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas (1.0.4).

  • (optional) I have confirmed this bug exists on the master branch of pandas (87247370c14629c1797fcf3540c6bd93b777a17e).


Code Sample, a copy-pastable example

import numpy as np
import pandas as pd

ser1 = pd.Series([1], dtype="Int64")
ser2 = pd.Series([1.0], dtype=np.float64)
ser1 += ser2

Note that the issue is NOT specific to the integer EA, but happens with all EAs that don't use NumPy dtypes.

Problem description

Traceback (most recent call last):
  File "bug.py", line 6, in <module>
    ser1 += ser2
  File ".../pandas/core/generic.py", line 11305, in __iadd__
    return self._inplace_method(other, type(self).__add__)  # type: ignore[operator]
  File ".../pandas/core/generic.py", line 11289, in _inplace_method
    if self.ndim == 1 and result._indexed_same(self) and result.dtype == self.dtype:
TypeError: Cannot interpret 'Int64Dtype()' as a data type

This is likely a fallout of #37508. See here:

pandas/pandas/core/generic.py

Lines 11283 to 11292 in 8724737

def _inplace_method(self, other, op):
"""
Wrap arithmetic method to operate inplace.
"""
result = op(self, other)
if self.ndim == 1 and result._indexed_same(self) and result.dtype == self.dtype:
# GH#36498 this inplace op can _actually_ be inplace.
self._values[:] = result._values
return self

There, the result dtype (which is a NumPy dtype in this case) is compared with the EA dtype. This raises the TypeError, because NumPy only allows comparing its dtype objects with "real" NumPy dtypes (and EAs don't provide NumPy-compatible dtypes).

Expected Output

Pass. I'm pretty sure that worked with 1.0.3 (the mentioned PR in question was backported to the 1.0.x branch between the 1.0.3 and 1.0.4 release).

@marco-neumann-by marco-neumann-by added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Nov 17, 2020
@jbrockmendel
Copy link
Member

best guess is this can be fixed by changing result.dtype == self.dtype to is_dtype_equal(result.dtype, self.dtype)

@jbrockmendel jbrockmendel reopened this Nov 17, 2020
@jorisvandenbossche jorisvandenbossche added ExtensionArray Extending pandas with custom dtypes or arrays. and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Nov 17, 2020
@jorisvandenbossche jorisvandenbossche added this to the Contributions Welcome milestone Nov 17, 2020
@jorisvandenbossche jorisvandenbossche added Regression Functionality that used to work in a prior pandas version and removed Bug labels Nov 17, 2020
@jorisvandenbossche
Copy link
Member

@marco-neumann-by Thanks for the report!

It seems we don't really have testing for inplace ops for EAs ..

@jorisvandenbossche jorisvandenbossche modified the milestones: Contributions Welcome, 1.1.5 Nov 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ExtensionArray Extending pandas with custom dtypes or arrays. Regression Functionality that used to work in a prior pandas version
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants