Skip to content

BUG: DataFrame.to_json does not take into account of time zone naive Timestamps #46112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
alanhowells opened this issue Feb 22, 2022 · 3 comments
Closed
2 of 3 tasks
Labels
Bug Duplicate Report Duplicate issue or pull request IO JSON read_json, to_json, json_normalize Timestamp pd.Timestamp and associated methods

Comments

@alanhowells
Copy link

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> import pandas as pd
>>> d = {'date': [pd.to_datetime('2021-02-18')], 'instant': [pd.to_datetime('2021-02-18T11:12:13')]}
>>> df = pd.DataFrame(data=d)
>>> df.to_csv()
',date,instant\r\n0,2021-02-18,2021-02-18 11:12:13\r\n'
>>> df.to_json(orient='table',index=False)
'{"schema":{"fields":[{"name":"date","type":"datetime"},{"name":"instant","type":"datetime"}],"pandas_version":"1.4.0"},"data":[{"date":"2021-02-18T00:00:00.000Z","instant":"2021-02-18T11:12:13.000Z"}]}'

Issue Description

When I give a date without a time or an datetime that is time zone naïve into a dataframe, the to_json result gives UTC time zone aware instants instead of time zone naïve instants.
to_csv does the right thing by only giving the date, and a time zone naïve instant.

2021-02-18,2021-02-18 11:12:13

to_json returns a UTC time zone aware instants for both the date and the instant.

{"date":"2021-02-18T00:00:00.000Z","instant":"2021-02-18T11:12:13.000Z"}

Expected Behavior

I would expect the to_json to return the following

>>> df.to_json(orient='table',index=False)
'{"schema":{"fields":[{"name":"date","type":"datetime"},{"name":"instant","type":"datetime"}],"pandas_version":"1.4.0"},"data":[{"date":"2021-02-18","instant":"2021-02-18T11:12:13.000"}]}'

Note the lack of time for the date and the lack of time zone designator for the instant.

{"date":"2021-02-18","instant":"2021-02-18T11:12:13.000"}

Installed Versions

INSTALLED VERSIONS
------------------
commit           : 06d230151e6f18fdb8139d09abf539867a8cd481
python           : 3.9.5.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.19042
machine          : AMD64
processor        : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : English_United Kingdom.1252

pandas : 1.4.1
numpy : 1.22.2
pytz : 2021.1
dateutil : 2.8.1
pip : 22.0.3
setuptools : 56.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

@alanhowells alanhowells added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Feb 22, 2022
@lithomas1
Copy link
Member

Is this a duplicate of #12997/#38760

@lithomas1 lithomas1 added Duplicate Report Duplicate issue or pull request IO JSON read_json, to_json, json_normalize Timestamp pd.Timestamp and associated methods and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Mar 1, 2022
@alanhowells
Copy link
Author

alanhowells commented Mar 1, 2022

@lithomas1 If DataFrame.to_json forwards to Series.to_json, then yes, this is a duplicate of #38760.

Sorry, I searched for "DataFrame" and "to_json", I should have searched for just "to_json".

Shall I close the issue?

@lithomas1
Copy link
Member

Yeah. I'll close it. Thanks for reporting anyways. I'll try to take a look sometime soonish as the tz-naive case doesn't seem terribly hard to fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Duplicate Report Duplicate issue or pull request IO JSON read_json, to_json, json_normalize Timestamp pd.Timestamp and associated methods
Projects
None yet
Development

No branches or pull requests

2 participants