Skip to content

QST: Python 3.9 testing in CI #36279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wumpus opened this issue Sep 10, 2020 · 11 comments
Closed

QST: Python 3.9 testing in CI #36279

wumpus opened this issue Sep 10, 2020 · 11 comments
Labels
Unicode Unicode strings
Milestone
1.1.3

Comments

@wumpus
Copy link

wumpus commented Sep 10, 2020

I use Travis CI to test, and I've tested all of my projects against the upcoming python 3.9 release. The only problem I'm seeing is dependent packages is pandas.

Pandas + 3.9 takes a long time to build and then dies with a compiler error. It also does this when I install cython first, which is the usual workaround when a new version of Python is close to release. Is some not-yet-mainstream cython version needed? I suppose there will be a wheel by the time 3.9.0 comes out in October?

Example CI log: https://travis-ci.com/github/wumpus/paramsurvey/jobs/383337409

  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  pandas/_libs/writers.c:4942:5: error: ‘PyUnicode_AsUnicode’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
   Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                               ^
  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  cc1: all warnings being treated as errors
  error: command '/usr/bin/gcc' failed with exit code 1

The logfile contains complete version information about the travisci environment.
@wumpus wumpus added Needs Triage Issue that has not been reviewed by a pandas team member Usage Question labels Sep 10, 2020
@jbrockmendel
Copy link
Member

@WillAyd is cython3 the blocker here?

@WillAyd
Copy link
Member

WillAyd commented Sep 11, 2020

I think the problem is we import PyUnicode_GET_SIZE in writers.pyx which is deprecated. Might be able to swap out with PyUnicode_GET_LENGTH assuming that is available in Cython

https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_GET_SIZE

@WillAyd WillAyd added good first issue and removed Needs Triage Issue that has not been reviewed by a pandas team member Usage Question labels Sep 11, 2020
@WillAyd
Copy link
Member

WillAyd commented Sep 11, 2020

@wumpus if you'd like to try I think you just need to update this and any occurrence within the module

pandas/pandas/_libs/writers.pyx

Line 5 in ddf2f05

from cpython.unicode cimport PyUnicode_GET_SIZE

@wumpus
Copy link
Author

wumpus commented Sep 11, 2020

I made that one change, and (no surprise) there are other calls to the "legacy" Unicode interface beyond that first one.

The deprecation plan is PEP 623 and py3.9 has the first deprecations.

I'm confused, though, why this isn't already fixed in pandas -- is -Wall turned off the the pandas CI build under py3.9?

@WillAyd
Copy link
Member

WillAyd commented Sep 11, 2020

We have the warnings turned on but they must not be throwing a deprecation warning until 3.9.

Can you post the full error list?

@wumpus
Copy link
Author

wumpus commented Sep 11, 2020 •
edited
Loading

Well, with just the one change to change GET_SIZE to GET_LENGTH (https://travis-ci.com/github/wumpus/pandas-appender/jobs/384012415) this is the next error:

  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -fPIC -DNPY_NO_DEPRECATED_API=0 -I/tmp/pip-build-env-589femm7/overlay/lib/python3.9/site-packages/numpy/core/include -I/home/travis/virtualenv/python3.9-dev/include -I/opt/python/3.9-dev/include/python3.9 -c pandas/_libs/writers.c -o build/temp.linux-x86_64-3.9/pandas/_libs/writers.o -Werror
  pandas/_libs/writers.c: In function ‘__pyx_f_6pandas_5_libs_7writers_word_len’:
  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  pandas/_libs/writers.c:4942:5: error: ‘PyUnicode_AsUnicode’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
   Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                               ^
  pandas/_libs/writers.c:4942:5: error: ‘_PyUnicode_get_wstr_length’ is deprecated [-Werror=deprecated-declarations]
       __pyx_v_l = PyUnicode_GET_SIZE(__pyx_v_val);
       ^
  In file included from /opt/python/3.9-dev/include/python3.9/unicodeobject.h:1026:0,
                   from /opt/python/3.9-dev/include/python3.9/Python.h:97,
                   from pandas/_libs/writers.c:33:
  /opt/python/3.9-dev/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
   static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                            ^
  cc1: all warnings being treated as errors
  error: command '/usr/bin/gcc' failed with exit code 1

PEP 623 appears to have a complete list of what was deprecated in py3.9. And yes, they were first deprecated in 3.9.

Edit: I looked at the Pandas travis CI looking for a successful 3.9 build, got tired of looking before I found one.

@WillAyd
Copy link
Member

WillAyd commented Sep 11, 2020

Strange - I made the edits above and it seemed to get rid of those warnings. I'll push up a PR soon

@WillAyd WillAyd mentioned this issue Sep 11, 2020
Closed
5 tasks
@wumpus
Copy link
Author

wumpus commented Sep 11, 2020

Well, I'm getting them with cython-0.29.21 (the current release), and the symbols aren't in pandas, they're inserted by cython.

@wumpus
Copy link
Author

wumpus commented Sep 12, 2020 •
edited
Loading

@WillAyd Looks like your PR built in Travis with cython-3, which is what @jbrockmendel asked you up top. With that clue, now I can try it for myself.

@wumpus
Copy link
Author

wumpus commented Sep 12, 2020 •
edited
Loading

... and that seems to have worked for my code to pass its tests. So I didn't run the pandas tests and my code doesn't use any Unicode, but indeed changing PyUnicode_GET_SIZE to _LENGTH and using Cython==3.0a6 seems to be a winning combination in python-3.9.0rc1

@simonjayhawkins simonjayhawkins added this to the 1.1.3 milestone Sep 13, 2020
@wumpus
Copy link
Author

wumpus commented Sep 16, 2020

Fixed by PR 36393

@wumpus wumpus closed this as completed Sep 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Unicode Unicode strings
Projects
None yet
Milestone
1.1.3
Development

Successfully merging a pull request may close this issue.

Py39 Support - Remove Deprecated Unicode Length Check WillAyd/pandas
4 participants
@WillAyd @wumpus @jbrockmendel @simonjayhawkins