Skip to content

BUG: Creating DataFrame frome Series with DatetimeIndex results in NaN-filled df #42118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
yohplala opened this issue Jun 19, 2021 · 3 comments
Closed
2 of 3 tasks
Labels
Closing Candidate May be closeable, needs more eyeballs DataFrame DataFrame data structure Usage Question

Comments

@yohplala
Copy link

yohplala commented Jun 19, 2021

  • I have checked that this issue has not already been reported (but not 100% sure either - search returns many tickets).

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Code Sample, a copy-pastable example

# Work when creating a df without datetime index.
pd.DataFrame({'a':pd.Series([1,2])})
Out[11]: 
   a
0  1
1  2

# Work when working with a datetime index and only with values of the Series
# (either with 'to_numpy()' or 'values').
pd.DataFrame({'a':pd.Series([1,2]).values},
             index=[pd.Timestamp('2021/01/01 08:00'), pd.Timestamp('2021/01/01 09:00')])
Out[12]: 
                     a
2021-01-01 08:00:00  1
2021-01-01 09:00:00  2

# Does not work when working directly with Series and datetime index.
pd.DataFrame({'a':pd.Series([1,2])},
             index=[pd.Timestamp('2021/01/01 08:00'), pd.Timestamp('2021/01/01 09:00')])
Out[13]: 
                      a
2021-01-01 08:00:00 NaN
2021-01-01 09:00:00 NaN

Problem description

In my real case problem, I am generating data through dict completion in which are handled pd.Series.
Something like:

var = pd.Series(list(range(1,5)))
data = { n : (var.shift(-n) - var)/var for n in range(1,3)}
df = pd.DataFrame(data, index=pd.date_range(pd.Timestamp('2021/01/01 08:00'), periods=4, freq='1h'))

And this results in a df filled with NaN

Is this a bug or a feature?

Expected Output

A df not filled with NaN.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.8.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.8.0-55-generic
Version : #62~20.04.1-Ubuntu SMP Wed Jun 2 08:55:04 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : fr_FR.UTF-8
LOCALE : fr_FR.UTF-8

pandas : 1.2.4
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.1
setuptools : 52.0.0.post20210125
Cython : 0.29.23
pytest : 6.2.3
hypothesis : None
sphinx : 4.0.1
blosc : None
feather : None
xlsxwriter : 1.3.8
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.0
IPython : 7.22.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 0.9.0
fastparquet : 0.6.3
gcsfs : None
matplotlib : 3.3.4
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : 2.0.0
pyxlsb : None
s3fs : None
scipy : 1.6.2
sqlalchemy : 1.4.15
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.51.2

@yohplala yohplala added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 19, 2021
@jreback
Copy link
Contributor

jreback commented Jun 19, 2021

when u create a frame from a series it is reindexed

this is a feature

@phofl
Copy link
Member

phofl commented Jun 19, 2021

Yes, this is the same as

df = pd.DataFrame({"a": pd.Series([1, 2])}, index=[3, 4])

@phofl phofl added Closing Candidate May be closeable, needs more eyeballs DataFrame DataFrame data structure Usage Question and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 19, 2021
@yohplala
Copy link
Author

Hi @jreback @phofl !
Thanks a lot for your reactivity. I feel this line makes it completely crystal clear

df = pd.DataFrame({"a": pd.Series([1, 2])}, index=[0,3,4])

df
Out[7]: 
     a
0  1.0
3  NaN
4  NaN

Sorry for the ticket then and thanks again!
Bests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Closing Candidate May be closeable, needs more eyeballs DataFrame DataFrame data structure Usage Question
Projects
None yet
Development

No branches or pull requests

3 participants