Skip to content

PERF: Speed up DatetimeConverter by using Matplotlib's epoch2num when possible... #6650

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 17, 2014
Merged

PERF: Speed up DatetimeConverter by using Matplotlib's epoch2num when possible... #6650

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1c65b96
Speed up DatetimeConverter by using Matplotlib's epoch2num when possi…
agijsberts Mar 15, 2014
dc95c4c
Added vbench performance benchmark for speed up of DatetimeConverter.
agijsberts Mar 16, 2014
35f1dea
Fixes to DatetimeConverter speed up following PR comments and corresp…
agijsberts Mar 17, 2014
64d9e92
Fix failed tests when importing Series from pandas.
agijsberts Mar 17, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion pandas/tseries/converter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,10 @@ def _dt_to_float_ordinal(dt):
preserving hours, minutes, seconds and microseconds. Return value
is a :func:`float`.
"""
base = dates.date2num(dt)
if isinstance(dt, np.ndarray) and com.is_datetime64_ns_dtype(dt):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ensures that the faster epoch2num variant is only used when appropriate (i.e., type(dt) = datetime64[ns]).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should test (np.ndarray,Series).

do dt.view('i8') instead of astyping (faster)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dt.view results in an AttributeError... can't figure out why.

>>> from pandas import date_range
>>> dt = date_range('2013-1-1', periods=100)
>>> dt.view('i8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arjan/tmp/git/pandas-agijsberts/pandas/core/index.py", line 190, in view
    result = super(Index, self).view(*args, **kwargs)
AttributeError: can't set attribute

dt.values.view('i8') instead works as expected, but I'd assume that also dt.view('i8') should work.

Edit: also dt.asi8 works. That seems to be the cleanest way perhaps.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's fine

base = dates.epoch2num(dt.astype(np.int64) / 1.0E9)
else:
base = dates.date2num(dt)
return base


Expand Down
42 changes: 42 additions & 0 deletions pandas/tseries/tests/test_converter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,11 @@
import nose

import numpy as np
from numpy.testing import assert_almost_equal as np_assert_almost_equal
from pandas import Timestamp
from pandas.compat import u
import pandas.util.testing as tm
from pandas.tseries.offsets import Second, Milli, Micro

try:
import pandas.tseries.converter as converter
Expand Down Expand Up @@ -46,9 +49,48 @@ def test_conversion(self):
rs = self.dtc.convert('2012-1-1', None, None)
self.assertEqual(rs, xp)

rs = self.dtc.convert(Timestamp('2012-1-1'), None, None)
self.assertEqual(rs, xp)

def test_conversion_float(self):
decimals = 9

rs = self.dtc.convert(Timestamp('2012-1-1 01:02:03', tz='UTC'), None, None)
xp = converter.dates.date2num(Timestamp('2012-1-1 01:02:03', tz='UTC'))
np_assert_almost_equal(rs, xp, decimals)

rs = self.dtc.convert(Timestamp('2012-1-1 09:02:03', tz='Asia/Hong_Kong'), None, None)
np_assert_almost_equal(rs, xp, decimals)

rs = self.dtc.convert(datetime(2012, 1, 1, 1, 2, 3), None, None)
np_assert_almost_equal(rs, xp, decimals)

def test_time_formatter(self):
self.tc(90000)

def test_dateindex_conversion(self):
decimals = 9

for freq in ('B', 'L', 'S'):
dateindex = tm.makeDateIndex(k = 10, freq = freq)
rs = self.dtc.convert(dateindex, None, None)
xp = converter.dates.date2num(dateindex)
np_assert_almost_equal(rs, xp, decimals)

def test_resolution(self):
def _assert_less(ts1, ts2):
val1 = self.dtc.convert(ts1, None, None)
val2 = self.dtc.convert(ts2, None, None)
if not val1 < val2:
raise AssertionError('{0} is not less than {1}.'.format(val1, val2))

# Matplotlib's time representation using floats cannot distinguish intervals smaller
# than ~10 microsecond in the common range of years.
ts = Timestamp('2012-1-1')
_assert_less(ts, ts + Second())
_assert_less(ts, ts + Milli())
_assert_less(ts, ts + Micro(50))


if __name__ == '__main__':
import nose
Expand Down
12 changes: 12 additions & 0 deletions vb_suite/timeseries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,3 +269,15 @@ def date_range(start=None, end=None, periods=None, freq=None):
dataframe_resample_max_numpy = \
Benchmark("df.resample('1s', how=np.max)", setup)


#----------------------------------------------------------------------
# DatetimeConverter

setup = common_setup + """
from pandas.tseries.converter import DatetimeConverter
"""

datetimeindex_converter = \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to define rng in the setup. pls post a run of this vbench as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be sure, rng is already defined in common_setup; it is best practice to redefine it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry....you are right..ok then

Benchmark('DatetimeConverter.convert(rng, None, None)',
setup, start_date=datetime(2013, 1, 1))