BLD: Compatibility issues with master branch cython #33372

ShaharNaveh · 2020-04-07T15:47:14Z

Continuation of issue #33224

With the latest version of cython, running python setup.py build_ext --inplace -j 0 gives the following errors:

`pandas/_libs/join.pyx`:

building 'pandas._libs.join' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/klib -I/home/user/Venvs/cythonM-pandas/lib/python3.8/site-packages/numpy/core/include -I/home/user/Venvs/cythonM-pandas/include -I/usr/include/python3.8 -c pandas/_libs/join.c -o build/temp.linux-x86_64-3.8/pandas/_libs/join.o
In file included from /usr/include/python3.8/tupleobject.h:41,
                 from /usr/include/python3.8/Python.h:105,
                 from pandas/_libs/join.c:32:
pandas/_libs/join.c: In function ‘__pyx_pf_6pandas_5_libs_4join_12inner_join_indexer.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/join.c:204503:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
204503 |             PyObject *r = PyTuple_GET_ITEM(o, n);
       |                           ^~~~~~~~~~~~~~~~
pandas/_libs/join.c: In function ‘__pyx_pf_6pandas_5_libs_4join_10left_join_indexer.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/join.c:204503:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
204503 |             PyObject *r = PyTuple_GET_ITEM(o, n);
       |                           ^~~~~~~~~~~~~~~~
pandas/_libs/join.c: In function ‘__pyx_pf_6pandas_5_libs_4join_8left_join_indexer_unique.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/join.c:204503:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
204503 |             PyObject *r = PyTuple_GET_ITEM(o, n);
       |                           ^~~~~~~~~~~~~~~~
pandas/_libs/join.c: In function ‘__pyx_pf_6pandas_5_libs_4join_14outer_join_indexer.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/join.c:204503:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
204503 |             PyObject *r = PyTuple_GET_ITEM(o, n);
       |                           ^~~~~~~~~~~~~~~~

`pandas/_libs/reshape.pyx`:

building 'pandas._libs.reshape' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 
-pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtun
e=generic -O3 -pipe -fno-plt -fPIC -DNPY_NO_DEPRECATED_API=0 -I/home/user/Venvs/cythonM-pandas/lib/python3.8/
site-packages/numpy/core/include -I/home/user/Venvs/cythonM-pandas/include -I/usr/include/python3.8 -c pandas
/_libs/reshape.c -o build/temp.linux-x86_64-3.8/pandas/_libs/reshape.o
In file included from /usr/include/python3.8/tupleobject.h:41,
                 from /usr/include/python3.8/Python.h:105,
                 from pandas/_libs/reshape.c:30:
pandas/_libs/reshape.c: In function ‘__pyx_pf_6pandas_5_libs_7reshape_unstack.isra.0’:
/usr/include/python3.8/cpython/tupleobject.h:27:60: warning: array subscript -1 is below array bounds of ‘PyObject *[1]’ {aka ‘struct _object *[1]’} [-Warray-bounds]
   27 | #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
      |                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
pandas/_libs/reshape.c:29728:27: note: in expansion of macro ‘PyTuple_GET_ITEM’
29728 |             PyObject *r = PyTuple_GET_ITEM(o, n);
      |                           ^~~~~~~~~~~~~~~~

join_zip.zip

reshape_zip.zip

Output of pd.show_versions():

INSTALLED VERSIONS

commit : bd91f45
python : 3.8.2.final.0
python-bits : 64
OS : Linux
OS-release : 5.5.13.b-1-hardened
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.0.dev0+1159.gbd91f4577
numpy : 1.18.2
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 46.1.3
Cython : 3.0a0 (https://github.com/cython/cython/tree/30c891e0dc36a553455df9064ecb1f13414a12d2)
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.13.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

The text was updated successfully, but these errors were encountered:

WillAyd · 2020-04-07T17:02:34Z

I couldn't reproduce this on master. Are you sure your environments are not getting crossed? pd.show_versions shows Python 3.7 yet the warnings are emitted from Python 3.8

ShaharNaveh · 2020-04-07T18:35:14Z

I couldn't reproduce this on master. Are you sure your environments are not getting crossed? pd.show_versions shows Python 3.7 yet the warnings are emitted from Python 3.8

You are completely right! that's because I couldn't install pandas on with cython-master, so I activated my conda environment to display the pd.show_versions()

ShaharNaveh · 2020-04-07T18:45:51Z

@WillAyd I "force" built the extensions, and I have managed to post the output of pd.show_versions() (Edited the OP)

WillAyd · 2020-04-07T21:41:37Z

Hmm somewhat confused by the responses - can you clarify that the environment you are using to cythonize the files is the same that you are using to compile things now?

ShaharNaveh · 2020-04-07T21:46:23Z

can you clarify that the environment you are using to cythonize the files is the same that you are using to compile things now?

@WillAyd Correct, I am cythonizing and compiling both on the same environment (python 3.8).

I created a separate environment for the cython-master, the cython-master environment is running on a python 3.8, while the "normal" pandas_-dev enviroment is running on python 3.7.

da-woods · 2020-04-07T22:01:27Z

@WillAyd It's definitely reproducible, and the warnings are pointing towards something that'll actually crash (so it isn't just a case of silencing warnings). I've submitted a fix for Cython so hopefully should be resolved soon. It's obvious exactly where it's coming from so doesn't need further diagnostics here I think

The Cython fused dispatch code was changed to make it faster, but that's clearly exposed a few other issues which need sorting.

WillAyd · 2020-04-07T22:09:47Z

Great! Thanks both of you

…

Sent from my iPhone

On Apr 7, 2020, at 3:01 PM, da-woods ***@***.***> wrote: @WillAyd It's definitely reproducible, and the warnings are pointing towards something that'll actually crash (so it isn't just a case of silencing warnings). I've submitted a fix for Cython so hopefully should be resolved soon. It's obvious exactly where it's coming from so doesn't need further diagnostics here I think The Cython fused dispatch code was changed to make it faster, but that's clearly exposed a few other issues which need sorting. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

da-woods · 2020-04-08T07:28:42Z

This should be fixed in Cython now.

ShaharNaveh · 2020-04-08T09:12:34Z

This should be fixed in Cython now.

I can confirmed that this is fixed!

Thank you @da-woods!

ShaharNaveh added Bug Needs Triage Issue that has not been reviewed by a pandas team member Build Library building on various platforms and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 7, 2020

This was referenced Apr 7, 2020

@wraparound(False) + @boundscheck(False) + fused types cython/cython#3492

Closed

Fused functions safe with wraparound+boundscheck off cython/cython#3493

Merged

ShaharNaveh closed this as completed Apr 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BLD: Compatibility issues with master branch cython #33372

BLD: Compatibility issues with master branch cython #33372

ShaharNaveh commented Apr 7, 2020 •

edited

Loading

INSTALLED VERSIONS

WillAyd commented Apr 7, 2020

ShaharNaveh commented Apr 7, 2020

ShaharNaveh commented Apr 7, 2020 •

edited

Loading

WillAyd commented Apr 7, 2020

ShaharNaveh commented Apr 7, 2020

da-woods commented Apr 7, 2020

WillAyd commented Apr 7, 2020 via email

da-woods commented Apr 8, 2020

ShaharNaveh commented Apr 8, 2020

BLD: Compatibility issues with master branch cython #33372

BLD: Compatibility issues with master branch cython #33372

Comments

ShaharNaveh commented Apr 7, 2020 • edited Loading

pandas/_libs/join.pyx:

pandas/_libs/reshape.pyx:

INSTALLED VERSIONS

WillAyd commented Apr 7, 2020

ShaharNaveh commented Apr 7, 2020

ShaharNaveh commented Apr 7, 2020 • edited Loading

WillAyd commented Apr 7, 2020

ShaharNaveh commented Apr 7, 2020

da-woods commented Apr 7, 2020

WillAyd commented Apr 7, 2020 via email

da-woods commented Apr 8, 2020

ShaharNaveh commented Apr 8, 2020

ShaharNaveh commented Apr 7, 2020 •

edited

Loading

`pandas/_libs/join.pyx`:

`pandas/_libs/reshape.pyx`:

ShaharNaveh commented Apr 7, 2020 •

edited

Loading