#18058: improve DatetimeIndex.date performance #18163

jamestran201 · 2017-11-08T04:06:11Z

closes PERF: making DatetimeIndex.date more performant #18058
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

jamestran201 · 2017-11-08T04:10:49Z

I ran asv with the tests from indexing.py, the output was "Benchmarks not significantly changed".

A quick test with %timeit:

In [1]: import pandas as pd

In [2]: rng = pd.date_range('2000-04-03', periods=200000, freq='2H')

In [3]: %timeit rng.date
555 ms ± 14.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit rng.date_new
90.4 ms ± 5.79 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]: %timeit rng.normalize().to_pydatetime()
121 ms ± 3.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [6]: rng.date
Out[6]:
array([datetime.date(2000, 4, 3), datetime.date(2000, 4, 3),
       datetime.date(2000, 4, 3), ..., datetime.date(2045, 11, 19),
       datetime.date(2045, 11, 19), datetime.date(2045, 11, 19)], dtype=object)

In [7]: rng.date_new
Out[7]:
array([datetime.date(2000, 4, 3), datetime.date(2000, 4, 3),
       datetime.date(2000, 4, 3), ..., datetime.date(2045, 11, 19),
       datetime.date(2045, 11, 19), datetime.date(2045, 11, 19)], dtype=object)

In [8]: rng.normalize().to_pydatetime()
Out[8]:
array([datetime.datetime(2000, 4, 3, 0, 0),
       datetime.datetime(2000, 4, 3, 0, 0),
       datetime.datetime(2000, 4, 3, 0, 0), ...,
       datetime.datetime(2045, 11, 19, 0, 0),
       datetime.datetime(2045, 11, 19, 0, 0),
       datetime.datetime(2045, 11, 19, 0, 0)], dtype=object)

Please let me know if there are some other asv tests that need to be run.

codecov · 2017-11-08T04:46:41Z

Codecov Report

Merging #18163 into master will decrease coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #18163      +/-   ##
==========================================
- Coverage   91.35%   91.33%   -0.03%     
==========================================
  Files         163      163              
  Lines       49714    49714              
==========================================
- Hits        45415    45405      -10     
- Misses       4299     4309      +10

Flag	Coverage Δ
#multiple	`89.13% <100%> (-0.01%)`	⬇️
#single	`39.63% <0%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/dtypes/concat.py	`99.13% <ø> (ø)`	⬆️
pandas/core/indexes/datetimes.py	`95.39% <100%> (-0.1%)`	⬇️
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/core/frame.py	`97.8% <0%> (-0.1%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 103ea6f...3b7fa77. Read the comment docs.

jreback · 2017-11-08T10:52:12Z

pandas/_libs/tslib.pyx


-    if tz is not None:


use the box param and have it take strings instead of boolean

iow box can be date/datetime/timestamp

you can assert that it is one of those

you will have to change places where this function. is called (eg box=True -> timestamp) etc

jreback · 2017-11-10T14:20:53Z

pandas/_libs/tslib.pyx

+
+def ints_to_pydatetime(ndarray[int64_t] arr, tz=None, freq=None,
+                       box="datetime"):
+    # convert an i8 repr to an ndarray of datetimes (if box == "datetime"),


can you add a Parameters section here

jreback · 2017-11-10T14:22:06Z

pandas/_libs/tslib.pyx

        func_create = create_datetime_from_ts

-    if tz is not None:
+    if tz is not None and box != "date":


instead of this, put an assert in the box == 'date' section that tz is None

if box == "date": assert (tz is not None), "tz should be None when converting to date" func_create = create_date_from_ts

@jreback Can I do something like this instead? That way the error message will be more specific when tz is used with date.

yes absolutely. The point is we should not be passing a tz with the date routine.

jreback · 2017-11-10T14:22:26Z

pandas/core/dtypes/concat.py

                x = x.reshape(shape)

        elif x.dtype == _TD_DTYPE:
            shape = x.shape
-            x = tslib.ints_to_pytimedelta(x.view(np.int64).ravel(), box=True)
+            x = tslib.ints_to_pytimedelta(x.view(np.int64).ravel(), box="True")


this is a different routine

jreback · 2017-11-11T13:22:01Z

pandas/_libs/tslib.pyx

        func_create = create_datetime_from_ts

-    if tz is not None:
+    if tz is not None and box != "date":


yes absolutely. The point is we should not be passing a tz with the date routine.

jreback · 2017-11-12T00:40:56Z

can u add an asv for this?

jreback · 2017-11-12T00:41:56Z

add a whatsnew note for 0.22, perf section

jamestran201 · 2017-11-12T04:14:37Z

Is the change in the timeseries.py file what you meant? Do you know how I can resolve the conflict?

jreback · 2017-11-12T15:42:45Z

asv_bench/benchmarks/timeseries.py

@@ -107,6 +107,8 @@ def time_infer_freq_daily(self):
    def time_infer_freq_business(self):
        infer_freq(self.b_freq)

+    def time_to_date(self):
+    	self.rng.date


might as well as ones for self.rng.time() and self.rng.to_pydatetime() while at it here.

note that this should be self.rng.date()

DatetimeIndex.date and DatetimeIndex.time are tagged with the @property tag at the moment. Do you want me to remove that tag so that they can be used as functions instead?

https://github.com/tmnhat2001/pandas/blob/e1b6d6230f54f3eee90225b25f8907989e3828df/pandas/core/indexes/datetimes.py#L1631

https://github.com/tmnhat2001/pandas/blob/e1b6d6230f54f3eee90225b25f8907989e3828df/pandas/core/indexes/datetimes.py#L1640

sorry that was a typo;

oh, it's ok

jreback · 2017-11-12T15:43:15Z

doc/source/whatsnew/v0.22.0.txt

@@ -69,8 +69,9 @@ Removal of prior version deprecations/changes
 Performance Improvements
 ~~~~~~~~~~~~~~~~~~~~~~~~

- Indexers on ``Series`` or ``DataFrame`` no longer create a reference cycle (:issue:`17956`)
+- Indexers on Series or DataFrame no longer create a reference cycle (:issue:`17956`)
 - Added a keyword argument, ``cache``, to :func:`to_datetime` that improved the performance of converting duplicate datetime arguments (:issue:`11665`)


should only include your change.

jreback · 2017-11-12T15:44:47Z

pandas/core/indexes/datetimes.py

@@ -1633,8 +1643,7 @@ def date(self):
        Returns numpy array of python datetime.date objects (namely, the date
        part of Timestamps without timezone information).
        """
-        return self._maybe_mask_results(libalgos.arrmap_object(
-            self.asobject.values, lambda x: x.date()))
+        return self.normalize().to_pydate()


just call the code for what you have in to_pydate() above, and remove that method; we don't actually want to expose this to the user.

jreback · 2017-11-12T15:46:41Z

doc/source/whatsnew/v0.22.0.txt

 - Added a keyword argument, ``cache``, to :func:`to_datetime` that improved the performance of converting duplicate datetime arguments (:issue:`11665`)
+- DatetimeIndex.date skips the iterations that create many python objects (:issue:`18058`)


Improved performance of :func:`Series.dt.date` and :func:`DatetimeIndex.date`

jreback

small comments. needs a rebase. can you show the updated asv results. (for those 3 that we added), you can do --bench timeseries

jreback · 2017-11-13T01:47:28Z

pandas/_libs/tslib.pyx

+    Parameters
+    ----------
+    arr  : array of i8 repr
+    tz   : the timezone to convert to,


so doc-strings should be slightly different

arr : array of i8 tz : str, default None convert to this timezone freq : str/Offset, default None freq to convert box : {'datetime', 'timestamp', 'date'}, default 'datetime' # add what you have below here

The asv results are:
asv continuous -f 1.1 upstream/master issue18058 -b timeseries.DatetimeIndex.time_dti_time -b timeseries.DatetimeIndex.time_to_date -b timeseries.DatetimeIndex.time_to_pydatetime

conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt ·· Installing into conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt · Running 6 total benchmarks (2 commits * 1 environments * 3 benchmarks) [ 0.00%] · For pandas commit hash 04c90d90: [ 0.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt [ 0.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt [ 16.67%] ··· Running timeseries.DatetimeIndex.time_dti_time 312ms [ 33.33%] ··· Running timeseries.DatetimeIndex.time_to_date 40.6ms [ 50.00%] ··· Running timeseries.DatetimeIndex.time_to_pydatetime 46.9ms [ 50.00%] · For pandas commit hash c176a3c2: [ 50.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt [ 50.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt [ 66.67%] ··· Running timeseries.DatetimeIndex.time_dti_time 375ms [ 83.33%] ··· Running timeseries.DatetimeIndex.time_to_date 258±4ms [100.00%] ··· Running timeseries.DatetimeIndex.time_to_pydatetime 48.8±1ms before after ratio [c176a3c2] [04c90d90] - 375ms 312ms 0.83 timeseries.DatetimeIndex.time_dti_time - 258±4ms 40.6ms 0.16 timeseries.DatetimeIndex.time_to_date

great!

we should fix time too :) (another PR)

ah, yeah, i was about to ask

jreback · 2017-11-13T03:19:42Z

can u rebase again
sorry lots of merges happening

jamestran201 · 2017-11-13T03:26:23Z

yeah, i'm on it

jamestran201 · 2017-11-13T03:36:49Z

just finished it, do i need to push again?

jreback · 2017-11-13T13:04:00Z

yes need to push again.

pep8speaks · 2017-11-14T00:52:28Z

Hello @tmnhat2001! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on November 22, 2017 at 04:46 Hours UTC

jreback · 2017-11-22T02:34:50Z

can you rebase

…etimeIndex

jreback · 2017-11-22T11:25:43Z

thanks @tmnhat2001

jreback · 2017-11-22T11:26:09Z

FYI @jbrockmendel IIRC you may have a PR to move some of this code to tslibs.conversion

jamestran201 · 2017-11-22T12:27:56Z

thanks @jreback
I'll open a pr for dti.time as well

jbrockmendel · 2017-11-22T15:33:29Z

IIRC you may have a PR to move some of this code to tslibs.conversion

@jreback #18389 is close, but shouldn't overlap with this.

jreback reviewed Nov 8, 2017

View reviewed changes

jreback added Performance Memory or execution speed performance Datetime Datetime data dtype labels Nov 8, 2017

jreback requested changes Nov 10, 2017

View reviewed changes

jreback requested changes Nov 11, 2017

View reviewed changes

jreback requested changes Nov 12, 2017

View reviewed changes

jreback reviewed Nov 12, 2017

View reviewed changes

jreback requested changes Nov 13, 2017

View reviewed changes

jamestran201-alt added 9 commits November 21, 2017 23:41

#18058: improve DatetimeIndex.date performance

cae4400

Fix PEP8 issue

1ec7216

Modify argument 'box' of ints_to_pydatetime

b0ca4ee

Add comments to ints_to_pydatetime and fix typo in concat.py

244c983

Fix assertion in ints_to_pydatetime

d932761

Update whatsnew for 0.22, add asv for DatetimeIndex.date

67982c7

Add asv for dti.time and dti.to_pydatetime, remove to_date() from Dat…

6a10a21

…etimeIndex

Fix doc string for ints_to_pydatetime

213b1c4

Fix PEP8 issue

3b7fa77

jreback added this to the 0.22.0 milestone Nov 22, 2017

jreback approved these changes Nov 22, 2017

View reviewed changes

jreback merged commit 9776868 into pandas-dev:master Nov 22, 2017

jamestran201 deleted the issue18058 branch November 24, 2017 00:48

This was referenced May 31, 2018

Datetime.index.date returns incorrect date post upgrade to version 0.23 #21230

Closed

BUG: DatetimeIndex.time incorrect in case of timezones #21267

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#18058: improve DatetimeIndex.date performance #18163

#18058: improve DatetimeIndex.date performance #18163

jamestran201 commented Nov 8, 2017 •

edited

Loading

jamestran201 commented Nov 8, 2017

codecov bot commented Nov 8, 2017 •

edited

Loading

jreback Nov 8, 2017

jreback Nov 10, 2017

jreback Nov 10, 2017

jamestran201 Nov 11, 2017 •

edited

Loading

jreback Nov 11, 2017

jreback Nov 10, 2017

jreback Nov 11, 2017

jreback commented Nov 12, 2017

jreback commented Nov 12, 2017

jamestran201 commented Nov 12, 2017

jreback Nov 12, 2017

jamestran201 Nov 12, 2017 •

edited

Loading

jreback Nov 12, 2017

jamestran201 Nov 12, 2017

jreback Nov 12, 2017

jreback Nov 12, 2017

jreback Nov 12, 2017

jreback left a comment

jreback Nov 13, 2017

jamestran201 Nov 13, 2017

jreback Nov 13, 2017

jamestran201 Nov 13, 2017

jreback commented Nov 13, 2017

jamestran201 commented Nov 13, 2017

jamestran201 commented Nov 13, 2017

jreback commented Nov 13, 2017

pep8speaks commented Nov 14, 2017 •

edited

Loading

jreback commented Nov 22, 2017

jreback commented Nov 22, 2017

jreback commented Nov 22, 2017

jamestran201 commented Nov 22, 2017

jbrockmendel commented Nov 22, 2017

		- Added a keyword argument, ``cache``, to :func:`to_datetime` that improved the performance of converting duplicate datetime arguments (:issue:`11665`)
		- DatetimeIndex.date skips the iterations that create many python objects (:issue:`18058`)

#18058: improve DatetimeIndex.date performance #18163

#18058: improve DatetimeIndex.date performance #18163

Conversation

jamestran201 commented Nov 8, 2017 • edited Loading

jamestran201 commented Nov 8, 2017

codecov bot commented Nov 8, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamestran201 Nov 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 12, 2017

jreback commented Nov 12, 2017

jamestran201 commented Nov 12, 2017

Choose a reason for hiding this comment

jamestran201 Nov 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 13, 2017

jamestran201 commented Nov 13, 2017

jamestran201 commented Nov 13, 2017

jreback commented Nov 13, 2017

pep8speaks commented Nov 14, 2017 • edited Loading

Comment last updated on November 22, 2017 at 04:46 Hours UTC

jreback commented Nov 22, 2017

jreback commented Nov 22, 2017

jreback commented Nov 22, 2017

jamestran201 commented Nov 22, 2017

jbrockmendel commented Nov 22, 2017

jamestran201 commented Nov 8, 2017 •

edited

Loading

codecov bot commented Nov 8, 2017 •

edited

Loading

jamestran201 Nov 11, 2017 •

edited

Loading

jamestran201 Nov 12, 2017 •

edited

Loading

pep8speaks commented Nov 14, 2017 •

edited

Loading