Skip to content

MNT: Mark all nogil functions as noexcept #53379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 30, 2023

Conversation

matusvalo
Copy link
Contributor

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Tests added and passed if fixing a bug or adding a new feature
  • All code checks passed.
  • Added type annotations to new arguments/methods/functions.
  • Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

This PR marks all nogil functions as noexcept because:

  1. Currently, they are not propagating exception with Cython 0.29.X
  2. I found out that several of functions are still checking exception after every call - e.g.
    cpdef iso_calendar_t get_iso_calendar(int year, int month, int day) nogil:
  3. Several functions returning void were missed in MNT: Explicitly set cdef functions as noexcept #53144

Note: nogil function calls checking the exception after every function call is introducing non-negligible performance degradation - see https://cython.readthedocs.io/en/latest/src/userguide/language_basics.html#error-return-values

@matusvalo
Copy link
Contributor Author

@h-vetinari could you run tests over this PR using Cython3.0 b3? Unittests are failing on my computer but I am not 100% convinced that it is caused by changes in this PR.

@h-vetinari
Copy link
Contributor

@h-vetinari could you run tests over this PR using Cython3.0 b3?

Running in conda-forge/pandas-feedstock#162

@h-vetinari
Copy link
Contributor

Ah, I should have said: I'm running the changes of this PR on top of 2.0.1. If we need everything that's in main as well, then I'll have to do it differently...

@h-vetinari
Copy link
Contributor

So, on top of 2.0.1, the only failure on unix is test_add_out_of_pydatetime_range, but that was pre-existing. Windows has three other failures, but those also were pre-existing.

@mroeschke mroeschke added Refactor Internal refactoring of code Internals Related to non-user accessible pandas implementation and removed Refactor Internal refactoring of code labels May 25, 2023
@WillAyd
Copy link
Member

WillAyd commented May 25, 2023

What does something being nogil have to do with its error handling? I think this is OK but not sure I follow the reasoning in the OP; want to make sure we don't lose exception handling we actually want

@h-vetinari
Copy link
Contributor

What does something being nogil have to do with its error handling?

It has nothing to do with nogil, but with Cython 3 now propagating exceptions by default, and it being beneficial to mark those functions that will not raise as noexcept for performance reasons.

@WillAyd
Copy link
Member

WillAyd commented May 26, 2023

Cool makes sense and thanks for the info. This lgtm - assuming we can look at non-nogil functions in a follow up?

@matusvalo
Copy link
Contributor Author

matusvalo commented May 28, 2023

What does something being nogil have to do with its error handling?

Let me explain it using following function from PR:

cdef void remove_sum(float64_t val, int64_t *nobs, float64_t *sum_x,
float64_t *compensation) nogil:

  1. The function has void return type which in Cython3 means that by default after every call the exception is checked. This single point is introducing a small (maybe negligible) performance hit. In Cython 0.29.X you did not care since the default behaviour is to not propagate exception information.
  2. But the gil keyword is making it worse when function is called within with nogil block - e.g. here:
    remove_sum(values[j], &nobs, &sum_x, &compensation_remove)

    In this case cython3 generates following code:
     __pyx_f_6pandas_5_libs_6window_12aggregations_remove_sum(/*function args*/); if (unlikely(__Pyx_ErrOccurredWithGIL())) __PYX_ERR(0, 167, __pyx_L4_error)
    The problem is caused by __Pyx_ErrOccurredWithGIL() which acquires GIL lock. This can hit performance seriously. See [BUG] Performance degredation due to __Pyx_ErrOccurredWithGIL in memoryview code cython/cython#5324 for details.

@mroeschke mroeschke added this to the 2.1 milestone May 30, 2023
@mroeschke mroeschke merged commit db9083c into pandas-dev:main May 30, 2023
@mroeschke
Copy link
Member

Thanks @matusvalo

topper-123 pushed a commit to topper-123/pandas that referenced this pull request Jun 5, 2023
* MNT: Mark all `nogil` functions as `noexcept`

* Make linter happy
Daquisu pushed a commit to Daquisu/pandas that referenced this pull request Jul 8, 2023
* MNT: Mark all `nogil` functions as `noexcept`

* Make linter happy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Internals Related to non-user accessible pandas implementation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants