-
-
Notifications
You must be signed in to change notification settings - Fork 18.4k
Fixed GIL regressions with Cython 3 #55650
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm seeing 4 more instances of nogil without noexcept, 3 in parsers.pyx and one in offsets.pyx. Add these as well?
Which functions are you looking at? In theory I think we only want to do this for non-integral return types. The integral returning functions should be checking the error sentinel first before trying to acquire the GIL, which I think is more reasonable than |
@@ -1605,7 +1605,7 @@ cdef _categorical_convert(parser_t *parser, int64_t col, | |||
|
|||
# -> ndarray[f'|S{width}'] | |||
cdef _to_fw_string(parser_t *parser, int64_t col, int64_t line_start, | |||
int64_t line_end, int64_t width): | |||
int64_t line_end, int64_t width) noexcept: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have to put noexcept even when there's no nogil?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whether or not the GIL is in play you will now be doing a PyErr_Occurred() after each invocation, which causes Python runtime interaction and will be "slower". With that said I didn't benchmark this particular function - just reset it to the original .29 behavior. I'll try to run some tests later though to see if it matters; it would be better to check for exceptions in case the numpy call fails, or this gets refactored in the future.
Generally I'd consider the use of void functions in our Cython code bad practice, unless failure within the function absolutely cannot be handled (ex: in destructors). It would be better to always return an int for error handling, which Cython can check without Python runtime interaction
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Will.
Thanks @WillAyd |
cdef int _try_double_nogil(parser_t *parser, | ||
float64_t (*double_converter)( | ||
const char *, char **, char, | ||
char, char, int, int *, int *) nogil, | ||
char, char, int, int *, int *) noexcept nogil, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on #55650 (comment), don't we not want noexcept here? It's still not clear to me why we don't though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good callout - this PR just got us back to 0.29 behavior but we could do better with the error handling. The problem is the body of the function returns 1 on error and doesn't set a Python exception. We at least would need to return -1 and I think set a Python exception in an ideal world
They were indeed all integral return types. Can you explain further the italic bit above? Is the "should be checking" bit done by the caller? Isn't whether an exception was raised checked before we get back to the caller? |
ref #55179
The main issue was that the functions returning
double
were not markednoexcept
, so in Cython 3.0 these now explicitly check the global Python error indicator, which requires a reacquisition of the GILThe changes to
to_fw_string
were to fix warnings generated by Cython that subsequently appeared:We maybe shouldn't just be swallowing errors like this, but that is a different problem for a different day...for now this should get us back to 0.29 performance