BUG: [pyarrow] Bizarre overflow error when subtracting two identical Index objects.

[randolf-scholz]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

datetimes = [None, "2022-01-01T10:00:30", "2022-01-01T10:01:00"]
dt = pd.Index(datetimes, dtype="timestamp[ms][pyarrow]")

offset = pd.Timestamp("2022-01-01 10:00:30")
unit = pd.Index([pd.Timedelta(30, "s")], dtype="duration[ms][pyarrow]").item()

# %% encode to double[pyarrow]
encoded = (dt - offset) / unit
decoded = (encoded.round().astype(float) * unit + offset).astype(dt.dtype)

# compare original and decoded
pd.testing.assert_index_equal(dt, decoded, exact=True)  # ✅
assert ((dt - dt) == 0).all()  # ✅
assert ((decoded - decoded) == 0).all()  # ✅
assert ((decoded - dt) == 0).all()  # ✅
assert ((dt - decoded) == 0).all()  # ❌ overflow ?!?!

Issue Description

This one is absolutely baffling to me. Two Index objects, despite satisfying assert_index_equal, raise an exception when subtracting each other. I guess assert_index_equal must be omitting some internal differences under the hood?

It occurs after encoding and decoding a timestamp[ms][pyarrow] index to floating and back to timestamp[ms][pyarrow], by subtracting and offset and dividing by some frequency.

Even more weird is that it makes a difference how we define the unit:

# %% Try with different units
unit_a = pd.Timedelta(30, "s")  # <-- this one works!
unit_b = pd.Index([pd.Timedelta(30, "s")], dtype="duration[ms][pyarrow]").item()
assert unit_a == unit_b  # ✅
assert hash(unit_a) == hash(unit_b)  # ✅

# encode to double[pyarrow]
encoded_a = (dt - offset) / unit_a
encoded_b = (dt - offset) / unit_b
pd.testing.assert_index_equal(encoded_a, encoded_b, exact=True)  # ✅

# decode
decoded_a = (encoded_a.round().astype(float) * unit_a + offset).astype(dt.dtype)
decoded_b = (encoded_b.round().astype(float) * unit_b + offset).astype(dt.dtype)
pd.testing.assert_index_equal(decoded_a, decoded_b, exact=True)  # ✅
pd.testing.assert_index_equal(dt, decoded_a, exact=True)  # ✅
pd.testing.assert_index_equal(dt, decoded_b, exact=True)  # ✅
assert ((dt - decoded_a) == 0).all()  # ✅
assert ((dt - decoded_b) == 0).all()  # ❌ overflow ?!?!

Moreover, manually computing the difference in pyarrow works as well:

# %% compute differences in pyarrow:
import pyarrow as pa

x = pa.Array.from_pandas(dt)
y = pa.Array.from_pandas(decoded)
pa.compute.subtract(x, y)  # ✅
pa.compute.subtract(y, x)  # ✅

Expected Behavior

Either assert_index_equal should show some discrepancy, or dt an decoded should behave interchangably.

Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.11.7.final.0
python-bits : 64
OS : Linux
OS-release : 6.5.0-41-generic
Version : #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 3 11:32:55 UTC 2
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.2.2
numpy : 2.0.0
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 70.1.0
pip : 24.1
Cython : None
pytest : 8.2.2
hypothesis : 6.103.5
sphinx : 7.3.7
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.4
IPython : 8.25.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.6.0
gcsfs : None
matplotlib : 3.9.0
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.4
pandas_gbq : None
pyarrow : 16.1.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.13.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[mroeschke]

Thanks for the report. pandas uses the _checked variety of the arithmetic ops, so this example also fails in pyarrow even thought I would expect this not to fail

In [9]: import pyarrow as pa
   ...: 
   ...: x = pa.Array.from_pandas(dt)
   ...: y = pa.Array.from_pandas(decoded)

In [11]: pa.compute.subtract_checked(x, y)
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
Cell In[11], line 1
----> 1 pa.compute.subtract_checked(x, y)

File /opt/miniconda3/envs/pandas-dev/lib/python3.10/site-packages/pyarrow/compute.py:246, in _make_generic_wrapper.<locals>.wrapper(memory_pool, *args)
    244 if args and isinstance(args[0], Expression):
    245     return Expression._call(func_name, list(args))
--> 246 return func.call(args, None, memory_pool)

File /opt/miniconda3/envs/pandas-dev/lib/python3.10/site-packages/pyarrow/_compute.pyx:385, in pyarrow._compute.Function.call()

File /opt/miniconda3/envs/pandas-dev/lib/python3.10/site-packages/pyarrow/error.pxi:154, in pyarrow.lib.pyarrow_internal_check_status()

File /opt/miniconda3/envs/pandas-dev/lib/python3.10/site-packages/pyarrow/error.pxi:91, in pyarrow.lib.check_status()

ArrowInvalid: overflow

In [17]: x
Out[17]: 
<pyarrow.lib.TimestampArray object at 0x150422440>
[
  null,
  2022-01-01 10:00:30.000,
  2022-01-01 10:01:00.000
]

In [18]: y
Out[18]: 
<pyarrow.lib.TimestampArray object at 0x151dc7d60>
[
  null,
  2022-01-01 10:00:30.000,
  2022-01-01 10:01:00.000
]

[randolf-scholz]

Here's a simpler repro directly starting from pyarrow, something seems to go wrong during casting:

from datetime import datetime, timedelta
import pyarrow as pa

timestamps = [None, datetime(2022, 1, 1, 10, 0, 30), datetime(2022, 1, 1, 10, 1, 0)]
x = pa.array(timestamps, type=pa.timestamp("ms"))
pa.compute.subtract_checked(x, x)  # ✅

# convert to pandas and back
y = pa.Array.from_pandas(x.to_pandas())
pa.compute.subtract_checked(y, x)  # ✅
pa.compute.subtract_checked(x, y)  # ❌ overflow

I'll cross post in the pyarrow git.

[mroeschke]

Closing for now since this is an upstream issue