Skip to content

direct conversions for dt64 units #19487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

jbrockmendel
Copy link
Member

ATM almost all datetimestruct conversions in tslibs use PANDAS_FR_ns. The exceptions are two functions in tslibs.conversions that convert non-nano datetime64 to datetime64[ns]. This PR replaces those two calls with a more performant implementation (asvs on the way).

The perf improvement is available b/c the functions currently convert dt64[not-ns] --> datetimestruct and then convert datetimestruct-->dt64[ns]. But for most units the conversion is just a multiplication, so we can cut out the convert-to-dtstruct step.

Finally np_datetime.convert_datetime_to_dtstruct is not actually used, but is included for exposition. After this PR, ujson notwithstanding (#19486), the remaining src/datetime files could be simplified quite a bit

@jbrockmendel
Copy link
Member Author

In [4]: arr = np.arange(1, 100000, dtype='datetime64[h]')
In [6]: func = pd._libs.tslibs.conversion.ensure_datetime64ns
In [7]: %timeit func(arr)

Master:

In [8]: %timeit func(arr)
100 loops, best of 3: 3.55 ms per loop

In [9]: %timeit func(arr)
100 loops, best of 3: 3.58 ms per loop

In [10]: %timeit func(arr)
100 loops, best of 3: 3.58 ms per loop

PR:

In [8]: %timeit func(arr)
100 loops, best of 3: 2.28 ms per loop

In [9]: %timeit func(arr)
100 loops, best of 3: 2.28 ms per loop

In [10]: %timeit func(arr)
100 loops, best of 3: 2.28 ms per loop

But the year is about 6x slower and the month currently has a bug (that only got exposed when profiling, so apparently there is no test) so closing for now.

@jbrockmendel jbrockmendel deleted the tslibs-units2 branch February 11, 2018 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant