Default ExtensionScalarOpsMixin for divmod is incorrect

[TomAugspurger]

Code Sample, a copy-pastable example if possible

In [1]: from pandas.tests.extension.decimal.array import *

In [2]: arr = DecimalArray([decimal.Decimal(1), decimal.Decimal(2), decimal.Decimal(3)])

In [3]: divmod(arr, 2)
Out[3]:
[(Decimal('0'), Decimal('1')),
 (Decimal('1'), Decimal('0')),
 (Decimal('1'), Decimal('1'))]

Expected Output

In [11]: a, b = zip(*(divmod(x, 2) for x in arr))

In [12]: DecimalArray(a), DecimalArray(b)
Out[12]:
(DecimalArray(array([Decimal('0'), Decimal('1'), Decimal('1')], dtype=object)),
 DecimalArray(array([Decimal('1'), Decimal('0'), Decimal('1')], dtype=object)))

Any others?

[TomAugspurger]

So in

pandas/pandas/core/arrays/base.py

Lines 777 to 782 in 5ce06b5

	if coerce_to_dtype:
	try:
	res = self._from_sequence(res)
	except TypeError:
	pass

we catch a TypeError. I suppose that's necessary, because the ops we're implementing may not be an algebra (the array may not be able to hold the return type).

@Dr-Irv unreatalted to this issue, we allocate an len(self) list in

pandas/pandas/core/arrays/base.py

Line 768 in 5ce06b5

ovalues = [param] * len(self)

. Could that just be itertools.cycle(param)?

[TomAugspurger]

Are there any other operators that return a tuple? rdivmod I suppose.

[Dr-Irv]

@TomAugspurger Been on vacation, so just seeing this.

In the first one, the idea is that we are allowing the result to be a different EA type, which may not be the EA type of the caller.

With respect to using itertools.cycle, I think that would work. I was unaware that existed.

[Dr-Irv]

Also, with respect to the divmod issue reported here, in pandas/tests/extension/decimal/test_decimal.py, I explicitly skipped the test for divmod because things like divmod(Decimal(5.2), Decimal(3.3)) are not necessarily well defined. (or at least that is what I remember concluding way back when)

[TomAugspurger]

Thanks. divmod between two decimals seems to work fine, so we'll re-enable that test with the implementation.

In the first one, the idea is that we are allowing the result to be a different EA type, which may not be the EA type of the caller.

I'm not sure I follow. The expected result of a divmod(array, Any) should be a tuple of arrays

In [15]: divmod(np.array([1, 2, 3]), 2)
Out[15]: (array([0, 1, 1]), array([1, 0, 1]))

right now we're returning an array (list) of tuples.

[Dr-Irv]

In terms of divmod between two decimals working fine, I think the issue is how the "infinite" precision of Decimal should handle it. Here's an example:

In [1]: from decimal import Decimal

In [2]: import numpy as np

In [3]: r=np.random.randn(2)

In [4]: r
Out[4]: array([-1.42964197, -0.4781252 ])

In [5]: divmod(r[0], r[1])
Out[5]: (2.0, -0.47339156548270145)

In [6]: divmod(Decimal(r[0]), Decimal(r[1]))
Out[6]: (Decimal('2'), Decimal('-0.4733915654827014485306335700'))

And then there is this example:

In [8]: divmod(Decimal(1.0), Decimal(1.0)/Decimal(3.0))
Out[8]: (Decimal('3'), Decimal('1E-28'))

I think the result of that should be (Decimal('3'), Decimal('0.0'))

So it is not clear what the expected result should be when the input values are "infinite" decimal numbers. But maybe the test will work now if enabled.

When I wrote:

In the first one, the idea is that we are allowing the result to be a different EA type, which may not be the EA type of the caller.

I meant the following. Suppose the EA holds elements of scalar type A. Suppose that the result of the binop for the two scalars of type A is of scalar type B. Then the someone might have a different EA type holding scalars of type B. With the current try/except logic, this case is then covered.

[TomAugspurger]

I think the result of that should be (Decimal('3'), Decimal('0.0'))

You think that's a bug in Python?

Regardless, the array version should be matching what the scalar version does.

Then the someone might have a different EA type holding scalars of type B. With the current try/except logic, this case is then covered

I don't see that. On master, we return a list if the self._from_sequence fails, not a different extension array.

[Dr-Irv]

I'm not sure if it is a bug in Python, but it makes writing a test for divmod in DecimalArray a challenge, so I punted.

I agree that the array version should do what the scalar version does.

I don't see that. On master, we return a list if the self._from_sequence fails, not a different extension array.

Yes, and that was intentional.

I was responding to your comment here:
#22930 (comment)
where you wrote:

we catch a TypeError. I suppose that's necessary, because the ops we're implementing may not be an algebra (the array may not be able to hold the return type).

So I think we're all good in terms of understanding this.