CI: Numpy Dev Build Failing

[alimcmaster1]

Looks like our numpy dev build also uses a pre-release (alpha) of cython.

Is this the intended behaviour?

Think this is potentially what is causing the test failures:
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=33245&view=logs&j=3a03f79d-0b41-5610-1aa4-b4a014d0bc70&t=4d05ed0e-1ed3-5bff-dd63-1e957f2766a9

attrs                     19.3.0                     py_0  
ca-certificates           2020.1.1                      0  
certifi                   2020.4.5.1               py37_0  
**cython                    3.0a1                    pypi_0    pypi**
execnet                   1.7.1                      py_0  
hypothesis                5.5.4                      py_0  
importlib_metadata        1.5.0                    py37_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
more-itertools            8.2.0                      py_0  
ncurses                   6.2                  he6710b0_0  
numpy                     1.19.0.dev0+8f7adad          pypi_0    pypi
openssl                   1.1.1f               h7b6447c_0  
packaging                 20.3                       py_0  
pandas                    1.1.0.dev0+1243.gd3ff12cff           dev_0    <develop>
pip                       20.0.2                   py37_1  
pluggy                    0.13.1                   py37_0  
py                        1.8.1                      py_0  
pyparsing                 2.4.6                      py_0  
pytest                    5.4.1                    py37_0  
pytest-azurepipelines     0.8.0                      py_0  
pytest-forked             1.1.3                      py_0  
pytest-xdist              1.31.0                     py_0  
python                    3.7.7           hcf32534_0_cpython  
**python-dateutil           2.8.2.dev25+gc175137          pypi_0    pypi**
pytz                      2019.3                     py_0  
readline                  8.0                  h7b6447c_0  
scipy                     1.5.0.dev0+f614064          pypi_0    pypi
setuptools                46.1.3                   py37_0  
six                       1.14.0                   py37_0  
sortedcontainers          2.1.0                    py37_0  
sqlite                    3.31.1               h7b6447c_0  
tk                        8.6.8                hbc83047_0  
wcwidth                   0.1.9                      py_0  
wheel                     0.34.2                   py37_0  
xz                        5.2.4                h14c3975_4  
zipp                      2.2.0                      py_0  
zlib                      1.2.11               h7b6447c_3

[TomAugspurger]

I think we want Cython dev to catch regressions there.

cc @scoder if you have any ideas why this would be raising.

>   @cython.wraparound(False)
E   TypeError: Expected dict, got bool

pandas/_libs/algos.pyx:799: TypeError

I'll try to recompile with Cython master to get the actual C code.

[TomAugspurger]

I'm not really sure what's helpful to post here from the C file. This is a fused type so there's tons of output. Haven't been able to make a simpler reproducer yet, and I need to step away for a while.

[scoder]

@TomAugspurger No idea. The error message is plain wrong and I don't remember a specific change in that corner. A quick look in the sources also didn't reveal anything suspicious. It should accept a bool, and that's it. (And that also works in our test suite, at least according to the test we have…)

[WillAyd]

This is as minimal as a reproducer as I;ve found so far:

from numpy cimport int64_t, float64_t

ctypedef fused int_or_float:
    int64_t
    float64_t


def afunc(int_or_float[:] data,
          ties_method="average",
          bint ascending=True,
          na_option="keep",
          bint pct=False):
    return 5

Not sure specifically within that what is throwing things off. Seems like changing any of the above makes it OK

[TomAugspurger]

Thanks. I'm bisecting Cython commits with that.

[TomAugspurger]

I may have messed up the bisect. It's pointing to cython/cython@f5ab2f4, which doesn't really make sense...

Based on the title cython/cython#3493 seems like a likely culprit, but checking out that commit it seems fine.

[TomAugspurger]

I won't be able to look into this more today. For now I'd recommend pinning Cython in the numpydev env.

[scoder]

It's pointing to cython/cython@f5ab2f4, which doesn't really make sense...

Maybe you didn't clean your Cython installation before the bisect? That commit changes the version number, which could enforce a new installation and ABI module.

[scoder]

@WillAyd how do you reproduce the failure with that code? I'm asking because it compiles without error for me. How do you get it to fail? Is there anything else that I need?

[WillAyd]

It’s a run time error not compile time. Not at a computer but if you call that with an ndarray it will throw (can provide concrete steps in a few hours if needed)

…

Sent from my iPhone

On Apr 13, 2020, at 1:28 PM, Stefan Behnel ***@***.***> wrote:  @WillAyd how do you reproduce the failure with that code? I'm asking because it compiles without error for me. How do you get it to fail? Is there anything else that I need? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[WillAyd]

If needed here's the runtime code:

>>> import numpy as np
>>> import test
>>> test.afunc(np.array([1, 5]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "test.pyx", line 8, in test.__pyx_fused_cpdef
    def afunc(int_or_float[:] data,
TypeError: Expected dict, got bool

setup.py

from setuptools import setup
from Cython.Build import cythonize

import pkg_resources

numpy_incl = pkg_resources.resource_filename("numpy", "core/include")

setup(
    name='Bug test',
    ext_modules=cythonize("test.pyx"),
    include_dirs=[numpy_incl],
)

[scoder]

Ok, got it. It's this bug that strikes again. The default argument of the specialised fused function gets accidentally typed as dict, because the default argument of the generated dispatch function is a dict. There's a spot somewhere in the pipeline where the type gets overwritten or is inherited somehow. Need to find it…

[scoder]

I created a Cython ticket for this:
cython/cython#3511

[scoder]

Could you give the latest Cython master branch a try? It has a fix.