PERF: Fix performance regression in read_csv when converting datetimes #52057
Conversation
| )._values | ||
| ) | ||
| if isinstance(result, DatetimeIndex): | ||
| arr = result.to_numpy() |
There was a problem hiding this comment.
do we know it is timezone-naive here?
There was a problem hiding this comment.
no tz-aware.
I guess you are referring to result._values._ndarray? Tried this first, but breaks tests
There was a problem hiding this comment.
its its tzaware then to_numpy() should convert to object, which i havent checked but assume we dont want here
There was a problem hiding this comment.
Yeah it does, but I am not too concerned by this since this is the same behavior as before. This keeps performance at least stable.
There was a problem hiding this comment.
I have to clarify my initial response: we can either be tz aware or naive, depends on the input
There was a problem hiding this comment.
@jbrockmendel ok with merging? We should get this into 2.0
…d_csv when converting datetimes
…in read_csv when converting datetimes) (#52278) Backport PR #52057: PERF: Fix performance regression in read_csv when converting datetimes Co-authored-by: Patrick Hoefler <[email protected]>
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.the read-only pr changed this to _values which returns a DatetimeArray instead of a ndarray, which caused a performance regression
https://asv-runner.github.io/asv-collection/pandas/#io.csv.ReadCSVConcatDatetime.time_read_csv