Skip to content

BUG: Series containing Bool and NaN/None values returns Object dtype #57119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
DarthKitten2130 opened this issue Jan 28, 2024 · 4 comments
Closed
3 tasks done
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member

Comments

@DarthKitten2130
Copy link
Contributor

DarthKitten2130 commented Jan 28, 2024

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

ser1 = pd.Series([True,False,True,False,None])

print(ser1.dtype)

Issue Description

The dtype returned from the Series object is the 'object' dtype. This issue also persists in DataFrames.

Expected Behavior

The dtype returned should have been a 'bool' dtype.

Installed Versions

INSTALLED VERSIONS

commit : f538741
python : 3.11.3.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22631
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.2.0
numpy : 1.26.3
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 69.0.3
pip : 23.3.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.4
qtpy : None
pyqt5 : None

@DarthKitten2130 DarthKitten2130 added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jan 28, 2024
@mroeschke
Copy link
Member

bool (numpy type) cannot hold NA values, so this is the expected behavior https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#values-considered-missing

@davetapley
Copy link
Contributor

@mroeschke could it not return boolean?

The page you linked to does call it out a little later on:

:class:`NA` for :class:`StringDtype`, :class:`Int64Dtype` (and other bit widths),
:class:`Float64Dtype`(and other bit widths), :class:`BooleanDtype` and :class:`ArrowDtype`.

@DarthKitten2130
Copy link
Contributor Author

Actually, all 3 of us are technically correct. The doc you just referred to mentions that the dtype has to come from pyarrow, and not numpy. If you want this functionality, you would need to replace your numpy types with pyarrow types. As pandas moves away from numpy and embraces pyarrow, this will probably become more standardised. Until then however, I wish there would be a clear distinction as to how the transition from numpy to pyarrow affects legacy code.

@mroeschke
Copy link
Member

The boolean type is explicitly opt-in still, so when dtype=None, the inferred type is np.bool_

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member
Projects
None yet
Development

No branches or pull requests

3 participants