Skip to content

Replace is_temporal checks with Cython version #55506

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 19, 2023

Conversation

WillAyd
Copy link
Member

@WillAyd WillAyd commented Oct 13, 2023

With Cython 3.0.3 I was unable to build pandas. Looks like is_timedelta / is_datetime functions were getting mixed between static / non-static declarations. I wasn't sure why that was happening, but while investigating noted that these got ported upstream by @jbrockmendel a few years back, so I think it makes sense to leverage that instead (also clears the issue)

@WillAyd WillAyd requested a review from MarcoGorelli as a code owner October 13, 2023 13:01
@@ -51,7 +52,7 @@ cdef inline int64_t get_nat() noexcept:
# --------------------------------------------------------------------
# Type Checking

cdef inline bint is_integer_object(object obj) noexcept nogil:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We previously declared is_timedelta64_object / is_datetime64_object to be nogil, which cascaded through the rest of the functions in this pxd file. However, the upstream function is not considered nogil, with some discussion coming back to here:

numpy/numpy#16266 (comment)

I think its a mistake for any of the functions in this module that touch CPython objects to be declared nogil, unless something more recent in CPython has declared these functions as such (haven't looked)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just rip this out since we're using the cnp version everywhere?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh never mind, i see you did that where relevant below

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are talking about the nogil right? Yea I did the minimum required for this PR. There are still other functions in util.pxd where it should probably be removed

@jbrockmendel
Copy link
Member

Does this affect our minimum cython/numpy versions? (IIRC cython has a numpy/__init__.pxd file that got used sometimes and they wanted to get rid of it but i dont know if that ever happened or what versions are affected)

@mroeschke mroeschke added the Internals Related to non-user accessible pandas implementation label Oct 13, 2023
@mroeschke
Copy link
Member

Would this allow us to unpin our Cython <3.0.3 in our CI files?

@jbrockmendel
Copy link
Member

As long as you're doing these, might as well change to use the other stuff upstreamed into numpy in numpy/numpy#16266?

@WillAyd
Copy link
Member Author

WillAyd commented Oct 13, 2023

Would this allow us to unpin our Cython <3.0.3 in our CI files?

Ah didn't realize we pinned that. I believe so; at least locally I can build the entire code base with this

@WillAyd
Copy link
Member Author

WillAyd commented Oct 13, 2023

As long as you're doing these, might as well change to use the other stuff upstreamed into numpy in numpy/numpy#16266?

Unfortunately there is a problem where the definition we have of NPY_DATETIMEUNIT in pandas._libs.tslibs.np_datetime differs from what Cython offers, the main difference being the Cython version does not offer NPY_FR_GENERIC as a value. Our code does some comparison to that value, so we can't swap over to using get_datetime64_unit because Cython/C will not be happy with the difference in those enum declarations

I think that's a bug in Cython - the source file numpy/ndarraytypes.h does have an enum member for NPY_FR_GENERIC so probably just needs to be upstreamed in Cython before we can fully implement your PR

@WillAyd
Copy link
Member Author

WillAyd commented Oct 13, 2023

@lithomas1
Copy link
Member

With Cython 3.0.3 I was unable to build pandas. Looks like is_timedelta / is_datetime functions were getting mixed between static / non-static declarations. I wasn't sure why that was happening, but while investigating noted that these got ported upstream by @jbrockmendel a few years back, so I think it makes sense to leverage that instead (also clears the issue)

This was a bug in Cython, but I'm happy to see that this fixes it.

@WillAyd
Copy link
Member Author

WillAyd commented Oct 17, 2023

OK I updated the enum definition upstream in numpy/numpy#24923 (comment) so can come back around for the rest of these when that goes through. Any concern with this PR in its current state?

@mroeschke
Copy link
Member

I'm OK with the fix. Could you also unpin Cython 3.0.3 in our files?

@lithomas1
Copy link
Member

#55582.

I think we can unpin generally anyways since Cython 3.0.4 is out.

@lithomas1 lithomas1 added this to the 2.2 milestone Oct 18, 2023
@mroeschke mroeschke merged commit ca39324 into pandas-dev:main Oct 19, 2023
@mroeschke
Copy link
Member

Thanks @WillAyd

@jbrockmendel
Copy link
Member

nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Internals Related to non-user accessible pandas implementation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants