PERF: Release GIL on some datetime ops #11263
Conversation
|
ohh nice! can u share some timings? |
| continue | ||
| out[i] = period_ordinal_to_dt64(periodarr[i], freq) | ||
| with nogil: | ||
| if periodarr[i] == NPY_NAT: |
There was a problem hiding this comment.
move nogil outside the loop
|
Here are some timings - getting a pretty nice speedup. In single-threaded case things are looking about flat. In [1]: from pandas.util.testing import test_parallel
In [2]: dti = pd.date_range('1900-1-1', periods=100000)
In [3]: def f():
...: for i in range(4):
...: dti.year
In [4]: @test_parallel(4)
...: def g():
...: dti.year
In [8]: %timeit f()
10 loops, best of 3: 25.8 ms per loop
In [9]: %timeit g()
100 loops, best of 3: 7.71 ms per loop |
| pandas_datetime_to_datetimestruct(dtindex[i], PANDAS_FR_ns, &dts) | ||
| out[i] = monthrange(dts.year, dts.month)[1] | ||
| pandas_datetime_to_datetimestruct(dtindex[i], PANDAS_FR_ns, &dts) | ||
| out[i] = days_per_month_table[is_leapyear(dts.year)][dts.month-1] |
There was a problem hiding this comment.
prob makes sense to define this as a c-function and make it nogil (the days_per_month......)
jreback
added
Datetime
| @@ -3849,6 +3849,7 @@ def get_time_micros(ndarray[int64_t] dtindex): | |||
|
|
|||
|
|
|||
| @cython.wraparound(False) | |||
| @cython.boundscheck(False) | |||
| def get_date_field(ndarray[int64_t] dtindex, object field): | |||
There was a problem hiding this comment.
If you declared field as char[:] instead would you be able to nogil the whole thing until raise?
There was a problem hiding this comment.
hmm, tried that out, but cython doesn't seem to take a view of strings like that? http://stackoverflow.com/questions/28203670/how-to-use-cython-typed-memoryviews-to-accept-strings-from-python
|
@chris-b1 loooks good. can you add a whatsnew note (perf) and squash. |
chris-b1
changed the title
|
@jreback - updated |
PERF: Release GIL on some datetime ops
|
thanks! |
|
@chris-b1 can you add these (clean then make again to see them) |
This is a WIP, but far enough along I thought I'd share and see if the approach was reasonable.
This releases the GIL on most vectorized field accessors (e.g.
dt.year) and conversion to and fromPeriod. May be places it could be done - obviously would be nice for parsing, but I'm not sure that's possible.