PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone #51247

MarcoGorelli · 2023-02-08T21:21:11Z

closes PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone #49048 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

trying to use https://peps.python.org/pep-0508/#environment-markers, let's see if this works

jbrockmendel · 2023-02-09T16:47:38Z

pyproject.toml

@@ -27,7 +27,8 @@ dependencies = [
  "numpy>=1.21.0; python_version>='3.10'",
  "numpy>=1.23.2; python_version>='3.11'",
  "python-dateutil>=2.8.2",
-  "pytz>=2020.1"
+  "pytz>=2020.1",
+  "tzdata>=2022.1; platform_system=='Windows'"


does this need to be checked in setup.py? or even just import it in timezones.pyx?

I honestly don't know, I couldn't reproduce this when building from source

Tempted to just ship it, and then check with the nightlies?

@lithomas1 thoughts?

LGTM. I'll tag this as build to run the wheel builders, which should sniff this stuff out.

@lithomas1 did you try this?

Yeah the wheel builder jobs ran on this PR. Looks like some failures. I don't have Windows access so I can't help more sorry.

the wheel builder jobs ran on this PR

Ah, I see now sorry - thanks! Taking a look

Actually, you might need a check for this in setup.py?

If you uninstall tzdata and try to run python setup.py develop, does this error correctly?

I don't know if setuptools reads this section of pyproject.toml.

giving this a go

Looks like this worked

In a new virtual environment on Windows, I:

pip installed numpy

pip installed cython

pip install versioneer[toml]

ran python setup.py develop

and tzdata got installed, along with python-dateutil. Here's (part of) my output:

copying build\lib.win-amd64-cpython-38\pandas\_libs\window\aggregations.cp38-win_amd64.pyd -> pandas\_libs\window copying build\lib.win-amd64-cpython-38\pandas\_libs\window\indexers.cp38-win_amd64.pyd -> pandas\_libs\window copying build\lib.win-amd64-cpython-38\pandas\_libs\writers.cp38-win_amd64.pyd -> pandas\_libs copying build\lib.win-amd64-cpython-38\pandas\io\sas\_sas.cp38-win_amd64.pyd -> pandas\io\sas copying build\lib.win-amd64-cpython-38\pandas\io\sas\_byteswap.cp38-win_amd64.pyd -> pandas\io\sas copying build\lib.win-amd64-cpython-38\pandas\_libs\json.cp38-win_amd64.pyd -> pandas\_libs Creating c:\users\user\pandas-dev\.venv\lib\site-packages\pandas.egg-link (link to .) Adding pandas 2.1.0.dev0+186.g4b054da685 to easy-install.pth file Installed c:\users\user\pandas-dev Processing dependencies for pandas==2.1.0.dev0+186.g4b054da685 Searching for tzdata>=2022.1 Reading https://pypi.org/simple/tzdata/ C:\Users\User\pandas-dev\.venv\lib\site-packages\pkg_resources\__init__.py:123: PkgResourcesDeprecationWarning: is an invalid version and will not be supported in a future release warnings.warn( Downloading https://files.pythonhosted.org/packages/fa/5e/f99a7df3ae2079211d31ec23b1d34380c7870c26e99159f6e422dcbab538/tzdata-2022.7-py2.py3-none-any.whl#sha256=2b88858b0e3120792a3c0635c23daf36a7d7eeeca657c323da299d2094402a0d Best match: tzdata 2022.7 Processing tzdata-2022.7-py2.py3-none-any.whl Installing tzdata-2022.7-py2.py3-none-any.whl to c:\users\user\pandas-dev\.venv\lib\site-packages Adding tzdata 2022.7 to easy-install.pth file Installed c:\users\user\pandas-dev\.venv\lib\site-packages\tzdata-2022.7-py3.8.egg

jbrockmendel · 2023-02-09T16:49:06Z

I'm on board in principle, dont know enough about pep508 to offer an informed opinion

jbrockmendel · 2023-03-02T00:15:21Z

@MarcoGorelli IIUC you added a warning in timezones.pyx to address the associated issue(s). do you want to do this in addition to that?

…indows

MarcoGorelli · 2023-03-02T08:48:27Z

@MarcoGorelli IIUC you added a warning in timezones.pyx to address the associated issue(s). do you want to do this in addition to that?

I don't think I did, no

…indows

MarcoGorelli · 2023-03-06T07:42:00Z

tzdata is versioned as 2022.1, 2022.2, ... on PyPI, but 2022a, 2022b on conda-forge 🤦

I can't see how to specify a minimum for both, but putting 2022.1 and then installing that dependency from PyPI seems to work

But I'm worried that if someone has already installed tzdata2022a from conda and then also installs tzdata2022.1 from PyPI then they'll get issues from the conflict. Or is this a non-issue?

I'm tempted to close, contact the tzdata conda-forge people and ask them to follow the same versioning as the PyPI package, and then specify a minimum which works for both

Removing from the 2.0.0 milestone anyway, this probably isn't something worth backporting (esp. if we're not sure about it)

jbrockmendel · 2023-03-09T16:29:16Z

ci/deps/actions-310-numpydev.yaml

  - pip:
    - "cython"
    - "--extra-index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple"
    - "--pre"
    - "numpy"
    - "scipy"
+    - "tzdata>=2022.1"


should these somehow be marked as windows-specific?

jbrockmendel · 2023-03-09T16:30:11Z

doc/source/whatsnew/v1.5.0.rst

@@ -657,7 +657,7 @@ Optional libraries below the lowest tested version may still work, but are not c
 +-----------------+-----------------+---------+
 | tabulate        |0.8.9            |    X    |
 +-----------------+-----------------+---------+
-| tzdata          |2022a            |         |
+| tzdata          |2022.1           |         |


do we still want to edit the 1.5 file?

MarcoGorelli · 2023-03-10T13:57:49Z

cool, I've gone with 2022.1 then, the conda recipe yaml file can specify 2022a

CI seems to be working fine now

MarcoGorelli · 2023-03-13T09:07:10Z

reckon we can get this in for the next release candidate?

MarcoGorelli · 2023-03-13T19:39:57Z

I tried this out in a fresh virtual environment on Windows, and tzdata got installed fine: here's my complete unedited traceback https://pastebin.com/JpuCz8PY

lithomas1 · 2023-03-13T20:27:20Z

ci/deps/actions-310.yaml

  - xarray>=0.21.0
  - xlrd>=2.0.1
  - xlsxwriter>=1.4.3
  - zstandard>=0.15.2
+
+  - pip:
+    - tzdata>=2022.1; platform_system=="Windows"


It's probably better to be consistent and go with the pip installed tzdata everywhere, so we don't get surprises if/when Github updates its runner images.

I guess there's a risk of things breaking for people without the tzdata package installed, but that's more of a zoneinfo/Python stdlib problem.

agree, but isn't that what I've done (i.e. pip-installed tzdata)?

I meant on all platforms not just Windows, sorry if I was unclear.

ok, sure, I've removed the windows-specific condition, and have installed/required it everywhere

pyproject.toml

…andas into require-tzdata-on-windows

lithomas1

LGTM. Let's give this a shot!

mroeschke

I think there's one more timezone extra reference in Installing optional dependencies with pip extras in whatsnew/v2.0.0.rst. Otherwise LGTM

mroeschke · 2023-03-15T15:27:22Z

Thanks @MarcoGorelli

lumberbot-app · 2023-03-15T15:27:49Z

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

git checkout 2.0.x
git pull

Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 3ea17805b072d575d1f55f50fe0027d0fdb9b985

You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #51247: PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone'

Push to a named branch:

git push YOURFORK 2.0.x:auto-backport-of-pr-51247-on-2.0.x

Create a PR against branch 2.0.x, I would have named this PR:

"Backport PR #51247 on branch 2.0.x (PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

phofl · 2023-03-17T09:37:09Z

@MarcoGorelli could you do the backport?

MarcoGorelli · 2023-03-17T09:43:24Z

yup, thanks! guess this didn't make the rc then 😄 oh well

…rom a list of Timestamp with timezone --------- Co-authored-by: MarcoGorelli <> (cherry picked from commit 3ea1780)

…ndex from a list of Timestamp with timezone) (#52045) Backport PR #51247: PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone --------- Co-authored-by: MarcoGorelli <> (cherry picked from commit 3ea1780)

MarcoGorelli force-pushed the require-tzdata-on-windows branch from 11bbbcc to b5ef6ae Compare February 8, 2023 21:27

MarcoGorelli added 2 commits February 9, 2023 10:58

require tzdata on windows

f4c9e0a

put minimum version?

3a9cc7d

MarcoGorelli force-pushed the require-tzdata-on-windows branch from d4b1791 to 3a9cc7d Compare February 9, 2023 11:02

MarcoGorelli added 2 commits February 9, 2023 11:22

try installing with pip in conda envs

a0ac092

fixup ci check

5a1983d

MarcoGorelli marked this pull request as ready for review February 9, 2023 11:49

MarcoGorelli requested review from jbrockmendel and mroeschke February 9, 2023 11:50

whatsnew

cafb5cc

jbrockmendel reviewed Feb 9, 2023

View reviewed changes

lithomas1 added the Build Library building on various platforms label Feb 10, 2023

Merge remote-tracking branch 'upstream/main' into require-tzdata-on-w…

9d6e2df

…indows

MarcoGorelli added this to the 2.0 milestone Mar 2, 2023

MarcoGorelli added 6 commits March 3, 2023 11:12

add tzdata to yaml files

c19dca9

Merge remote-tracking branch 'upstream/main' into require-tzdata-on-w…

4a29800

…indows

add tzdata>=2022.1 to test_wheels_windows.bat

1aaa8d0

try using OR statement

fa3f2bc

try another syntax

b3c006e

try using the pypi install

3a561be

MarcoGorelli marked this pull request as draft March 3, 2023 16:14

MarcoGorelli marked this pull request as ready for review March 6, 2023 07:39

MarcoGorelli removed this from the 2.0 milestone Mar 6, 2023

MarcoGorelli mentioned this pull request Mar 9, 2023

RLS: 2.0 #46776

Closed

1 task

jbrockmendel reviewed Mar 9, 2023

View reviewed changes

MarcoGorelli added 2 commits March 10, 2023 12:09

clarify what tzdata is neeeded for, remove it as optional

249f345

remove optional section on timezones

4b054da

lithomas1 added this to the 2.0 milestone Mar 13, 2023

Merge branch 'main' into require-tzdata-on-windows

fc5a2d4

lithomas1 reviewed Mar 13, 2023

View reviewed changes

mroeschke reviewed Mar 13, 2023

View reviewed changes

pyproject.toml Show resolved Hide resolved

MarcoGorelli added 3 commits March 14, 2023 09:32

require tzdata everywhere

51d44f4

Merge branch 'require-tzdata-on-windows' of github.com:MarcoGorelli/p…

d07b2ec

…andas into require-tzdata-on-windows

remove timezone extra in package-checks

624e0e7

lithomas1 approved these changes Mar 14, 2023

View reviewed changes

mroeschke reviewed Mar 14, 2023

View reviewed changes

remove final timezone extra reference

860338f

mroeschke approved these changes Mar 15, 2023

View reviewed changes

mroeschke merged commit 3ea1780 into pandas-dev:main Mar 15, 2023

lumberbot-app bot added the Still Needs Manual Backport label Mar 15, 2023

phofl removed the Still Needs Manual Backport label Mar 17, 2023

phofl mentioned this pull request Mar 17, 2023

Backport PR #51247 on branch 2.0.x (PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone) #52045

Merged

jbrockmendel mentioned this pull request Mar 30, 2023

PERF: objects_to_datetime64ns performance degradation for datetime columns #50537

Closed

3 tasks

MarcoGorelli mentioned this pull request Apr 3, 2023

pandas v2.0.0 conda-forge/pandas-feedstock#159

Merged

4 tasks

asishm-wk mentioned this pull request Aug 11, 2023

DOC: updated required dependencies list #54493

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone #51247

PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone #51247

MarcoGorelli commented Feb 8, 2023 •

edited

Loading

jbrockmendel Feb 9, 2023

MarcoGorelli Feb 9, 2023

jbrockmendel Feb 9, 2023

lithomas1 Feb 10, 2023

MarcoGorelli Mar 2, 2023

lithomas1 Mar 2, 2023

MarcoGorelli Mar 3, 2023

lithomas1 Mar 13, 2023

MarcoGorelli Mar 13, 2023

MarcoGorelli Mar 13, 2023

jbrockmendel commented Feb 9, 2023

jbrockmendel commented Mar 2, 2023

MarcoGorelli commented Mar 2, 2023

MarcoGorelli commented Mar 6, 2023 •

edited

Loading

jbrockmendel Mar 9, 2023

jbrockmendel Mar 9, 2023

MarcoGorelli commented Mar 10, 2023

MarcoGorelli commented Mar 13, 2023

MarcoGorelli commented Mar 13, 2023

lithomas1 Mar 13, 2023 •

edited

Loading

MarcoGorelli Mar 13, 2023

lithomas1 Mar 13, 2023

MarcoGorelli Mar 14, 2023

lithomas1 left a comment

mroeschke left a comment

mroeschke commented Mar 15, 2023

lumberbot-app bot commented Mar 15, 2023

phofl commented Mar 17, 2023

MarcoGorelli commented Mar 17, 2023 •

edited

Loading

PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone #51247

PERF: Construction of a DatetimeIndex from a list of Timestamp with timezone #51247

Conversation

MarcoGorelli commented Feb 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Feb 9, 2023

jbrockmendel commented Mar 2, 2023

MarcoGorelli commented Mar 2, 2023

MarcoGorelli commented Mar 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoGorelli commented Mar 10, 2023

MarcoGorelli commented Mar 13, 2023

MarcoGorelli commented Mar 13, 2023

lithomas1 Mar 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lithomas1 left a comment

Choose a reason for hiding this comment

mroeschke left a comment

Choose a reason for hiding this comment

mroeschke commented Mar 15, 2023

lumberbot-app bot commented Mar 15, 2023

phofl commented Mar 17, 2023

MarcoGorelli commented Mar 17, 2023 • edited Loading

MarcoGorelli commented Feb 8, 2023 •

edited

Loading

MarcoGorelli commented Mar 6, 2023 •

edited

Loading

lithomas1 Mar 13, 2023 •

edited

Loading

MarcoGorelli commented Mar 17, 2023 •

edited

Loading