BUG: inconsistent return types from __getitem__ vs iteration

[gboeing]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy as np
import pandas as pd

print(np.__version__)  # 2.0.2
print(pd.__version__)  # 2.2.3

data = pd.Series([333, 555])

# accessing scalar via __getitem__ returns <class 'numpy.int64'>
print(type(data[0]))

# accessing scalar via iteration returns <class 'int'>
print(type(next(iter(data))))

Issue Description

numpy 2.0 recently changed its representation of scalars to include type information. However, pandas produces inconsistent return types when one is accessing scalars with __getitem__ vs iterating over items, as demonstrated in the example code snippet.

This inconsistency is showing up in downstream projects like NetworkX: networkx/networkx#7763 (comment)

Expected Behavior

pandas should produce consistent return types when one is accessing scalars with __getitem__ vs iterating over items

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.12.8
python-bits : 64
OS : Linux
OS-release : 6.8.0-50-generic
Version : #51-Ubuntu SMP PREEMPT_DYNAMIC Sat Nov 9 17:58:29 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.3
numpy : 2.0.2
pytz : 2024.1
dateutil : 2.9.0.post0
pip : 24.3.1
Cython : None
sphinx : 8.1.3
IPython : 8.30.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : 1.4.2
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : 5.3.0
matplotlib : 3.9.3
numba : 0.60.0
numexpr : 2.10.2
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : 8.3.4
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : 0.23.0
tzdata : 2024.2
qtpy : None
pyqt5 : None

[gboeing]

#59838 is loosely related.

[rhshadrach]

Thanks for the report, somewhat related to #25969.

I also thought we had an issue as whether to return Python or NumPy scalars across the API, but can't seem to find it. I'm not sure whether there has been any prior discussions on this, cc @pandas-dev/pandas-core.

I'll also add there is inconsistency in reductions. On NumPy dtypes (e.g. int64):

NumPy scalars:

• sum
• mean
• prod
• quantile
• sem
• count

Python scalars:

• median
• min
• max
• kurt
• skew
• nunique

On PyArrow dtypes (e.g int64[pyarrow]), these are all Python scalars with the exception of count which gives a NumPy scalar.

[gboeing]

Thanks @rhshadrach. It's been a couple months so I just wanted to check back on this to see if any further information is needed?

[Dr-Irv]

This is a pretty old feature. Here's the release note about it in v0.21.0, from over 7 years ago:

https://pandas.pydata.org/docs/whatsnew/v0.21.0.html#iteration-of-series-index-will-now-return-python-scalars

With numpy 1.26.4, we get the following with the example code:

>>> import numpy as np
>>> import pandas as pd
>>>
>>> print(np.__version__)  # 2.0.2
1.26.4
>>> print(pd.__version__)  # 2.2.3
2.2.2
>>>
>>> data = pd.Series([333, 555])
>>>
>>> # accessing scalar via __getitem__ returns <class 'numpy.int64'>
>>> print(type(data[0]))
<class 'numpy.int64'>
>>>
>>> # accessing scalar via iteration returns <class 'int'>
>>> print(type(next(iter(data))))
<class 'int'>

So with older versions of numpy, our implementation of iter() matched theirs - we returned a python scalar. Now it's a question whether we go back. Definitely needs discussion.

[rhshadrach]

our implementation of iter() matched theirs - we returned a python scalar. Now it's a question whether we go back. Definitely needs discussion.

I would prefer if all pandas operations that returned scalars returned Python, rather than NumPy objects. This is especially true if we are wanting to make migration to arrow-backed dtypes easier.

[rhshadrach]

cc @pandas-dev/pandas-core