Skip to content

BUG (2.0rc1): Missing breaking API change in the changelog related to new timedelta64 dtypes #51965

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
aberres opened this issue Mar 14, 2023 · 1 comment
Closed
2 of 3 tasks
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member

Comments

@aberres
Copy link
Contributor

aberres commented Mar 14, 2023

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

x_start = pd.Series(pd.Timestamp(2023,3,14,12))
x_end = pd.Series(pd.Timestamp(2023,3,14,14))

# Pandas 1.5.3
# 0    7200000.0
# dtype: float64
# 
# Pandas 2.0.0rc0
# 0   0 days 02:00:00
# dtype: timedelta64[ms]
(x_end - x_start).astype("timedelta64[ms]")

Issue Description

When test-driving the RC we noticed that generating Plotly graphs failed (snippet for reference below).

When investigating, we found the following in the Plotly code: (x_end - x_start).astype("timedelta64[ms]"). With Pandas < 2 this returned float64 representing the ms between the timestamps.

Now that Pandas 2 has a timedelta64[ms] dtype this dtype is used instead.

When existing code expects a float64 dtype it might break - as in the Plotly case.

As workaround, the following works with all Pandas versions:

(x_end - x_start).astype("timedelta64[ns]") / pd.Timedelta(1, 'ms')`

Maybe it makes sense to add this change to the list of breaking changes?

It might also be worth to mention that casting from np.datetime64[D] results in timedelta64[s] now as noticed here.

Plotly reproduction

import plotly.express as px
import plotly.graph_objects as go
import pandas as pd

df = pd.DataFrame([{"Task": "Job A", "Start": "2022-01-01", "Finish": "2022-01-02"}])
fig_1 = px.timeline(df, x_start="Start", x_end="Finish", y="Task")
fig_1.update_yaxes(autorange="reversed")

Leads to

[...]
File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py:258, in JSONEncoder.iterencode(self, o, _one_shot)
    253 else:
    254     _iterencode = _make_iterencode(
    255         markers, self.default, _encoder, self.indent, floatstr,
    256         self.key_separator, self.item_separator, self.sort_keys,
    257         self.skipkeys, _one_shot)
--> 258 return _iterencode(o, 0)

File ~/venv/weplan3.11/lib/python3.11/site-packages/_plotly_utils/utils.py:136, in PlotlyJSONEncoder.default(self, obj)
    134     except NotEncodable:
    135         pass
--> 136 return _json.JSONEncoder.default(self, obj)

File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py:180, in JSONEncoder.default(self, o)
    161 def default(self, o):
    162     """Implement this method in a subclass such that it returns
    163     a serializable object for ``o``, or calls the base implementation
    164     (to raise a ``TypeError``).
   (...)
    178 
    179     """
--> 180     raise TypeError(f'Object of type {o.__class__.__name__} '
    181                     f'is not JSON serializable')

TypeError: Object of type timedelta is not JSON serializable

Expected Behavior

Changelog mentions the new behavior.

Installed Versions

@aberres aberres added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Mar 14, 2023
@aberres
Copy link
Contributor Author

aberres commented Mar 14, 2023

Ok, ok. I was blind. A colleague pointed me to the correct section.

image

@aberres aberres closed this as completed Mar 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member
Projects
None yet
Development

No branches or pull requests

1 participant