BUG: Debugging problems in Python 3.10 due to use of old Cython

[cartisan]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy.core.numeric as numeric
import pandas as pd
df = pd.DataFrame()  # Without this there is no crash
dtype = numeric.dtype(float)
print(dtype) # Just for run output

Issue Description

Executing this code works as expected. But when I place a PyCharm debugger breakpoint on any line after dtype = numeric.dtype(float) and run in PyCharm debug mode I get the following exception:

Traceback (most recent call last):
  File "/home/leonid/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/221.5787.24/plugins/python/helpers/pydev/pydevd.py", line 1491, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/leonid/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/221.5787.24/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/user/code/some_dir/some_dir/test.py", line 7, in <module>
    dtype = numeric.dtype(float)
TypeError: 'NoneType' object is not callable

This behavior has been observed in numpy ( numpy/numpy#21008 ) and determined to be related to a Cython issue. The issue has been resolved by the latest Cython release (0.29.30) and the newest numpy release (1.22.4) that was built using that Cython version seems to be doing fine.

Would it be possible to expedite a minor pandas version update, completely similar to 1.4.2 but built using the new Cython? This has been a major pain point for ML development at my company for quite some time, so I wanted to check in and see if perhaps anything can be done.

Expected Behavior

I can use the debugger on the above code.

Installed Versions

Traceback (most recent call last):
File "", line 1, in
File "/home/leonid/.local/share/virtualenvs/prism_ada-nMYNssaU/lib/python3.10/site-packages/pandas/util/_print_versions.py", line 109, in show_versions
deps = _get_dependency_info()
File "/home/leonid/.local/share/virtualenvs/prism_ada-nMYNssaU/lib/python3.10/site-packages/pandas/util/_print_versions.py", line 88, in _get_dependency_info
mod = import_optional_dependency(modname, errors="ignore")
File "/home/leonid/.local/share/virtualenvs/prism_ada-nMYNssaU/lib/python3.10/site-packages/pandas/compat/_optional.py", line 138, in import_optional_dependency
module = importlib.import_module(name)
File "/home/leonid/.pyenv/versions/3.10.5/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/leonid/.local/share/virtualenvs/prism_ada-nMYNssaU/lib/python3.10/site-packages/setuptools/init.py", line 8, in
import _distutils_hack.override # noqa: F401
File "/home/leonid/.local/share/virtualenvs/prism_ada-nMYNssaU/lib/python3.10/site-packages/_distutils_hack/override.py", line 1, in
import('_distutils_hack').do_override()
File "/home/leonid/.local/share/virtualenvs/prism_ada-nMYNssaU/lib/python3.10/site-packages/_distutils_hack/init.py", line 72, in do_override
ensure_local_distutils()
File "/home/leonid/.local/share/virtualenvs/prism_ada-nMYNssaU/lib/python3.10/site-packages/_distutils_hack/init.py", line 59, in ensure_local_distutils
assert '_distutils' in core.file, core.file
AssertionError: /home/leonid/.pyenv/versions/3.10.5/lib/python3.10/distutils/core.py

[phofl]

Hi, thanks for your report. We are already using the newest cython version. You can track the next release at #46610

[cartisan]

Hi Patrick, thanks for your quick response. So pandas 1.4.3 will be built with the newest Cython version, that's a great relief. :)

From what I understand, 1.4.3 was scheduled for the end of April but still faces some unresolved challenges, so it is unclear when it will be released. I was wondering if it is an option for pandas to release a stable version in the meantime, that fixes the debugging issue I described. Since you seem to be already building using the newest Cython, this should not involve much manual work? It would certainly be very appreciated (at least by me, but I gather there are quite some more affected in the community).

[jreback]

@cartisan releases will happen when they happen
we r blocked in any issue which folks are trying to resolve