Are tests failing on Windows?

[topper-123]

Running the tests on windows fail:

>>> pytest pandas\tests\tseries\offsets\test_offsets.py -x
――――――――――――――――――――― TestCommon.test_apply[BusinessDay0] ―――――――――――――――――――――

self = <pandas.tests.tseries.offsets.test_offsets.TestCommon object at 0x066D4530>
offset_types = <class 'pandas.tseries.offsets.BusinessDay'>

    def test_apply(self, offset_types):
        sdt = datetime(2011, 1, 1, 9, 0)
        ndt = np_datetime64_compat('2011-01-01 09:00Z')

        for dt in [sdt, ndt]:
            expected = self.expecteds[offset_types.__name__]
            self._check_offsetfunc_works(offset_types, 'apply', dt, expected)

            expected = Timestamp(expected.date())
            self._check_offsetfunc_works(offset_types, 'apply', dt, expected,
>                                        normalize=True)

pandas\tests\tseries\offsets\test_offsets.py:357:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas\tests\tseries\offsets\test_offsets.py:320: in _check_offsetfunc_works
    result = func(dt_tz)
pandas\tseries\offsets.py:85: in wrapper
    result = result.normalize()
pandas\_libs\tslibs\timestamps.pyx:1086: in pandas._libs.tslibs.timestamps.Timestamp.normalize
    normalized_value = normalize_i8_timestamps(
pandas\_libs\tslibs\conversion.pyx:1095: in pandas._libs.tslibs.conversion.normalize_i8_timestamps
    result = _normalize_local(stamps, tz)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   pos = trans.searchsorted(stamps, side='right') - 1
E   ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'int'

pandas\_libs\tslibs\conversion.pyx:1151: ValueError

Several tests fail.

By going back in commit history until the tests pass, it looks like this is caused by #21826.

[jschendel]

Have you tried rebuilding the cython extensions? Windows tests look good on appveyor. Can you post the output of pd.show_versions()?

[topper-123]

I build pandas using python setup.py build_ext --inplace -j 4. This works without failures. Is there an additional process for building extensions?

Output of pd.show_versions()

INSTALLED VERSIONS

commit: 365eac4
python: 3.6.3.final.0
python-bits: 32
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.21.0.dev+2313.g365eac4d2
pytest: 3.3.1
pip: 10.0.1
setuptools: 38.2.5
Cython: 0.28.3
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: 0.10.0
IPython: 6.4.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.9
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: None
bs4: None
html5lib: 1.0b10
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[jschendel]

That should work for building the extensions, provided that you've done it since pulling the commit you've linked to as the potential offender.

Looking over pd.show_versions(), the only thing that immediately stands out to me is "python-bits: 32". I don't think 32bit is very well tested by CI, and the error in question is complaining about expecting int64_t. Not an expert here though.

The other thing that looks a little weird is "pandas: 0.21.0.dev+2313.g365eac4d2", but I think that's just a superficial tags issue and unrelated to the failure; doing a git fetch --tags should make this 0.24.0dev.

cc @jbrockmendel @chris-b1 : would appreciate your thoughts on this

[topper-123]

I solved the tag issue by doing git pull upstream master --tags, so pd.version is now '0.24.0.dev0+313.g2b51c968c'. That didn't solve the issue.

Tried cloning pandas into a new directory. Same failure...

[chris-b1]

This is a 32 bit bug, which we don't have CI for, so looks like some of the casting from #21826 was necessary

[jbrockmendel]

so looks like some of the casting from #21826 was necessary

That's plausible. Should I revert I undo that part of the cleanup?

[xhochy]

This also fails in the Linux nigthlies: MacPython/pandas-wheels#32 / https://travis-ci.org/MacPython/pandas-wheels/builds/404071029?utm_source=github_status&utm_medium=notification

[jbrockmendel]

@topper-123 hoping to get your help to debug this. Can you edit the line:

pos = trans.searchsorted(stamps, side='right') - 1

to

pos = trans.searchsorted(stamps, side='right') - <int64_t>1

and see if that fixes the problem?

[topper-123]

Still a problem:

――――――――――――――――――――――― test_fy5253qtr_onoffset_nearest ―――――――――――――――――――――――

    def test_fy5253qtr_onoffset_nearest():
        # GH#19036
        ts = Timestamp('1985-09-02 23:57:46.232550356-0300',
                       tz='Atlantic/Bermuda')
        offset = FY5253Quarter(n=3, qtr_with_extra_week=1, startingMonth=2,
                               variation="nearest", weekday=0)
>       fast = offset.onOffset(ts)

pandas\tests\tseries\offsets\test_fiscal.py:645:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas\tseries\offsets.py:2018: in onOffset
    qtr_lens = self.get_weeks(dt)
pandas\tseries\offsets.py:1992: in get_weeks
    year_has_extra_week = self.year_has_extra_week(dt)
pandas\tseries\offsets.py:2002: in year_has_extra_week
    norm = Timestamp(dt).normalize().tz_localize(None)
pandas\_libs\tslibs\timestamps.pyx:1095: in pandas._libs.tslibs.timestamps.Timestamp.normalize
    normalized_value = normalize_i8_timestamps(
pandas\_libs\tslibs\conversion.pyx:1094: in pandas._libs.tslibs.conversion.normalize_i8_timestamps
    result = _normalize_local(stamps, tz)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   pos = trans.searchsorted(stamps, side='right') - <int64_t>1
E   ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'int'

pandas\_libs\tslibs\conversion.pyx:1150: ValueError

EDIT: BTW, I changed conversion.pyx:1150, a search for pos = trans.searchsorted(stamps, side='right') - 1 showed several locations. Should I change it in all locations?

[jbrockmendel]

Should I change it in all locations?

Before #21826 only one of the pos = trans.searchsorted(... has the dtype check, not sure why the others would be different.

One more thing I'd like to try if you're up for it:

pos = <ndarray[int64_t]>trans.searchsorted(stamps, side='right') - <int64_t>1

[chris-b1]

@jbrockmendel - you have pos typed as int64_t - searchsorted isn't guaranteed to return return that - instead it will return a signed pointer size type (NPY_INTP, equivalently, Py_ssize_t).

So you could change the type of pos to Py_ssize_t, which is what you want anyways, since it is an indexer.

[jbrockmendel]

@topper-123 can you try with #21923 and see if that fixes it?