Skip to content

BUG/COMPAT: Series bool type not promoted to object when introducing NaNs #13034

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
domspad opened this issue Apr 29, 2016 · 2 comments · Fixed by #41431
Closed

BUG/COMPAT: Series bool type not promoted to object when introducing NaNs #13034

domspad opened this issue Apr 29, 2016 · 2 comments · Fixed by #41431
Labels
good first issue Needs Tests Unit test(s) needed to prevent regressions
Milestone

Comments

@domspad
Copy link

domspad commented Apr 29, 2016

Code Sample, a copy-pastable example if possible

In [2]: import pandas as pd

In [3]: a = pd.Series([True,False,True])

In [4]: a
Out[4]: 
0     True
1    False
2     True
dtype: bool

In [6]: a[0] = pd.np.nan

In [7]: a
Out[7]: 
0    NaN
1    0.0
2    1.0
dtype: float64

Expected Output

based on type promotions mentioned here I would expect the series to get promoted to object to allow for NaNs.

When introducing NAs into an existing Series or DataFrame via reindex or some other means, boolean and integer types will be promoted to a different dtype in order to store the NAs. These are summarized by this table:

In [7]: a
Out[7]: 
0      NaN
1    False
2     True
dtype: object

output of pd.show_versions()

In [2]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Darwin
OS-release: 15.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US

pandas: 0.18.0
nose: 1.3.7
pip: 8.1.1
setuptools: 20.6.7
Cython: None
numpy: 1.10.4
scipy: 0.17.0
statsmodels: None
xarray: None
IPython: 4.1.2
sphinx: 1.3.5
patsy: None
dateutil: 2.5.2
pytz: 2016.3
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.39.0```
@jreback
Copy link
Contributor

jreback commented Apr 30, 2016

yeah that should coerce to object. Right now we are promoting a boolean with nulls -> float64

this is somewhat deep in there. need to look inside core/internals/Block.setitem. where _maybe_promote is called. Bools have not gotten much love lately.

Though to be fair we do this similar to how we do ints with NA's, meaning we cast to floats.

So not sure how much this is relied upon in the real world.

@jreback jreback added Bug Dtype Conversions Unexpected or buggy dtype conversions Difficulty Advanced labels Apr 30, 2016
@jreback jreback added this to the 0.19.0 milestone Apr 30, 2016
@jreback jreback changed the title Series bool type not promoted to object when introducing NaNs BUG/COMPAT: Series bool type not promoted to object when introducing NaNs Apr 30, 2016
@jreback jreback added the Compat pandas objects compatability with Numpy or Python functions label Apr 30, 2016
@jreback jreback modified the milestones: 0.20.0, Next Major Release Mar 23, 2017
@mroeschke mroeschke removed the Compat pandas objects compatability with Numpy or Python functions label Apr 10, 2020
@mroeschke
Copy link
Member

This looks to work on master now. Could use a test

In [3]: In [2]: import pandas as pd
   ...:
   ...: In [3]: a = pd.Series([True,False,True])

In [4]: a[0] = np.nan

In [5]: a
Out[5]:
0      NaN
1    False
2     True
dtype: object

@mroeschke mroeschke added good first issue Needs Tests Unit test(s) needed to prevent regressions and removed Bug Dtype Conversions Unexpected or buggy dtype conversions labels Apr 24, 2021
@jreback jreback modified the milestones: Contributions Welcome, 1.3 May 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Needs Tests Unit test(s) needed to prevent regressions
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants