Skip to content

BUG: shift() on Series of dtype bool makes dtype into object, so HDFStore won't append #4618

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bigtonylewis opened this issue Aug 21, 2013 · 2 comments
Labels
Dtype Conversions Unexpected or buggy dtype conversions Duplicate Report Duplicate issue or pull request Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Milestone

Comments

@bigtonylewis
Copy link

import pandas as pd

# '0.11.0'
print pd.__version__

df = pd.DataFrame(index=range(0,5), columns=['mybool'])
df.mybool = False
store = pd.HDFStore('shiftedbool.h5')

# this works
store.append('noshiftedbool', df)

# now add a shifted bool, backfilled
df['shiftedbool'] = df.mybool.shift(1).fillna(method='bfill')
print df

# causes exception:
store.append('shiftedbool', df)

This can be worked around; for now I'm using 0.0 and 1.0. But this seems like an issue that could be improved.

My first issue submitted; I hope it's up to scratch

@jreback
Copy link
Contributor

jreback commented Aug 21, 2013

#4610 fixes this ; the backfilled arry doesn't downcast (to bool dtype) correctly
the exception is correct by HDFstore

another workaround is to do this:

In [1]: s = Series([True],list(range(0,5)))

In [2]: s
Out[2]: 
0    True
1    True
2    True
3    True
4    True
dtype: bool

In [3]: s.shift(1)
Out[3]: 
0     NaN
1    True
2    True
3    True
4    True
dtype: object

In [4]: s.shift(1).fillna(False)
Out[4]: 
0    False
1     True
2     True
3     True
4     True
dtype: object

In [5]: s.shift(1).fillna(False).convert_objects()
Out[5]: 
0    False
1     True
2     True
3     True
4     True
dtype: bool

astype(bool) will also work here rather than convert_objects

@jreback
Copy link
Contributor

jreback commented Aug 21, 2013

closed by #4610

@jreback jreback closed this as completed Aug 21, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Dtype Conversions Unexpected or buggy dtype conversions Duplicate Report Duplicate issue or pull request Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Projects
None yet
Development

No branches or pull requests

2 participants