`test_maybe_promote_int_with_int` error on 32bit platform #31856

toddrme2178 · 2020-02-10T20:59:13Z

The test test_maybe_promote_int_with_int in tests/dtypes/cast/test_promote.py is failing for a subset of combinations where int8, uint8, int16, or uint16 values are promotoed to int32 or uint32 targets on i586 the platform (but not on x86_64). The error message is pretty consistent.

This failure is new in pandas 1.0.1, it was not present on 1.0.0.

The specific combination of values that fail are:

int16-32768-int32
int8-32768-int32
uint16--1-int32
uint16--129-int32
uint16-2147483648-uint32
uint16-65536-uint32
uint8-2147483648-uint32
uint8-65536-uint32

These do not fail:

int16--32769-int32
int8--32769-int32
uint16--32769-int32
uint8--32769-int32

A specific example of the error message is:

______________ test_maybe_promote_int_with_int[uint16--129-int32] ______________
linux -- Python 3.7.3 /usr/bin/python3

dtype = dtype('uint16'), fill_value = -129, expected_dtype = dtype('int32')

    @pytest.mark.parametrize(
        "dtype, fill_value, expected_dtype",
        [
            # size 8
            ("int8", 1, "int8"),
            ("int8", np.iinfo("int8").max + 1, "int16"),
            ("int8", np.iinfo("int16").max + 1, "int32"),
            ("int8", np.iinfo("int32").max + 1, "int64"),
            ("int8", np.iinfo("int64").max + 1, "object"),
            ("int8", -1, "int8"),
            ("int8", np.iinfo("int8").min - 1, "int16"),
            ("int8", np.iinfo("int16").min - 1, "int32"),
            ("int8", np.iinfo("int32").min - 1, "int64"),
            ("int8", np.iinfo("int64").min - 1, "object"),
            # keep signed-ness as long as possible
            ("uint8", 1, "uint8"),
            ("uint8", np.iinfo("int8").max + 1, "uint8"),
            ("uint8", np.iinfo("uint8").max + 1, "uint16"),
            ("uint8", np.iinfo("int16").max + 1, "uint16"),
            ("uint8", np.iinfo("uint16").max + 1, "uint32"),
            ("uint8", np.iinfo("int32").max + 1, "uint32"),
            ("uint8", np.iinfo("uint32").max + 1, "uint64"),
            ("uint8", np.iinfo("int64").max + 1, "uint64"),
            ("uint8", np.iinfo("uint64").max + 1, "object"),
            # max of uint8 cannot be contained in int8
            ("uint8", -1, "int16"),
            ("uint8", np.iinfo("int8").min - 1, "int16"),
            ("uint8", np.iinfo("int16").min - 1, "int32"),
            ("uint8", np.iinfo("int32").min - 1, "int64"),
            ("uint8", np.iinfo("int64").min - 1, "object"),
            # size 16
            ("int16", 1, "int16"),
            ("int16", np.iinfo("int8").max + 1, "int16"),
            ("int16", np.iinfo("int16").max + 1, "int32"),
            ("int16", np.iinfo("int32").max + 1, "int64"),
            ("int16", np.iinfo("int64").max + 1, "object"),
            ("int16", -1, "int16"),
            ("int16", np.iinfo("int8").min - 1, "int16"),
            ("int16", np.iinfo("int16").min - 1, "int32"),
            ("int16", np.iinfo("int32").min - 1, "int64"),
            ("int16", np.iinfo("int64").min - 1, "object"),
            ("uint16", 1, "uint16"),
            ("uint16", np.iinfo("int8").max + 1, "uint16"),
            ("uint16", np.iinfo("uint8").max + 1, "uint16"),
            ("uint16", np.iinfo("int16").max + 1, "uint16"),
            ("uint16", np.iinfo("uint16").max + 1, "uint32"),
            ("uint16", np.iinfo("int32").max + 1, "uint32"),
            ("uint16", np.iinfo("uint32").max + 1, "uint64"),
            ("uint16", np.iinfo("int64").max + 1, "uint64"),
            ("uint16", np.iinfo("uint64").max + 1, "object"),
            ("uint16", -1, "int32"),
            ("uint16", np.iinfo("int8").min - 1, "int32"),
            ("uint16", np.iinfo("int16").min - 1, "int32"),
            ("uint16", np.iinfo("int32").min - 1, "int64"),
            ("uint16", np.iinfo("int64").min - 1, "object"),
            # size 32
            ("int32", 1, "int32"),
            ("int32", np.iinfo("int8").max + 1, "int32"),
            ("int32", np.iinfo("int16").max + 1, "int32"),
            ("int32", np.iinfo("int32").max + 1, "int64"),
            ("int32", np.iinfo("int64").max + 1, "object"),
            ("int32", -1, "int32"),
            ("int32", np.iinfo("int8").min - 1, "int32"),
            ("int32", np.iinfo("int16").min - 1, "int32"),
            ("int32", np.iinfo("int32").min - 1, "int64"),
            ("int32", np.iinfo("int64").min - 1, "object"),
            ("uint32", 1, "uint32"),
            ("uint32", np.iinfo("int8").max + 1, "uint32"),
            ("uint32", np.iinfo("uint8").max + 1, "uint32"),
            ("uint32", np.iinfo("int16").max + 1, "uint32"),
            ("uint32", np.iinfo("uint16").max + 1, "uint32"),
            ("uint32", np.iinfo("int32").max + 1, "uint32"),
            ("uint32", np.iinfo("uint32").max + 1, "uint64"),
            ("uint32", np.iinfo("int64").max + 1, "uint64"),
            ("uint32", np.iinfo("uint64").max + 1, "object"),
            ("uint32", -1, "int64"),
            ("uint32", np.iinfo("int8").min - 1, "int64"),
            ("uint32", np.iinfo("int16").min - 1, "int64"),
            ("uint32", np.iinfo("int32").min - 1, "int64"),
            ("uint32", np.iinfo("int64").min - 1, "object"),
            # size 64
            ("int64", 1, "int64"),
            ("int64", np.iinfo("int8").max + 1, "int64"),
            ("int64", np.iinfo("int16").max + 1, "int64"),
            ("int64", np.iinfo("int32").max + 1, "int64"),
            ("int64", np.iinfo("int64").max + 1, "object"),
            ("int64", -1, "int64"),
            ("int64", np.iinfo("int8").min - 1, "int64"),
            ("int64", np.iinfo("int16").min - 1, "int64"),
            ("int64", np.iinfo("int32").min - 1, "int64"),
            ("int64", np.iinfo("int64").min - 1, "object"),
            ("uint64", 1, "uint64"),
            ("uint64", np.iinfo("int8").max + 1, "uint64"),
            ("uint64", np.iinfo("uint8").max + 1, "uint64"),
            ("uint64", np.iinfo("int16").max + 1, "uint64"),
            ("uint64", np.iinfo("uint16").max + 1, "uint64"),
            ("uint64", np.iinfo("int32").max + 1, "uint64"),
            ("uint64", np.iinfo("uint32").max + 1, "uint64"),
            ("uint64", np.iinfo("int64").max + 1, "uint64"),
            ("uint64", np.iinfo("uint64").max + 1, "object"),
            ("uint64", -1, "object"),
            ("uint64", np.iinfo("int8").min - 1, "object"),
            ("uint64", np.iinfo("int16").min - 1, "object"),
            ("uint64", np.iinfo("int32").min - 1, "object"),
            ("uint64", np.iinfo("int64").min - 1, "object"),
        ],
    )
    def test_maybe_promote_int_with_int(dtype, fill_value, expected_dtype):
        dtype = np.dtype(dtype)
        expected_dtype = np.dtype(expected_dtype)
    
        # output is not a generic int, but corresponds to expected_dtype
        exp_val_for_scalar = np.array([fill_value], dtype=expected_dtype)[0]
    
>       _check_promote(dtype, fill_value, expected_dtype, exp_val_for_scalar)

../../BUILDROOT/python-pandas-1.0.1-58.1.i386/usr/lib/python3.7/site-packages/pandas/tests/dtypes/cast/test_promote.py:237: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../BUILDROOT/python-pandas-1.0.1-58.1.i386/usr/lib/python3.7/site-packages/pandas/tests/dtypes/cast/test_promote.py:94: in _check_promote
    _assert_match(result_fill_value, expected_fill_value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

result_fill_value = -129, expected_fill_value = -129

    def _assert_match(result_fill_value, expected_fill_value):
        # GH#23982/25425 require the same type in addition to equality/NA-ness
        res_type = type(result_fill_value)
        ex_type = type(expected_fill_value)
        if res_type.__name__ == "uint64":
            # No idea why, but these (sometimes) do not compare as equal
            assert ex_type.__name__ == "uint64"
        elif res_type.__name__ == "ulonglong":
            # On some builds we get this instead of np.uint64
            # Note: cant check res_type.dtype.itemsize directly on numpy 1.18
            assert res_type(0).itemsize == 8
            assert ex_type == res_type or ex_type == np.uint64
        else:
            # On some builds, type comparison fails, e.g. np.int32 != np.int32
>           assert res_type == ex_type or res_type.__name__ == ex_type.__name__
E           AssertionError: assert (<class 'numpy.intc'> == <class 'numpy.int32'>
E             -<class 'numpy.intc'>
E             +<class 'numpy.int32'> or 'intc' == 'int32'
E             - intc
E             + int32)

The error message is consistent, except that intc and int32 are replaced with uintc and uint32 in some cases. I can provide all error messages if desired.

The tests are being run with numpy 1.18.1.

The text was updated successfully, but these errors were encountered:

TomAugspurger · 2020-02-10T22:03:47Z

Thanks for the report. Are you able to bisect to the commit that caused the regression?

TomAugspurger · 2020-02-19T22:10:57Z

I'm updating our wheel-building infra for the new uploading stuff, and noticed these failures on 32-bit windows and linux jobs.

https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=29023&view=logs&j=dcd50a69-7bd8-59f7-72b1-a119e6ffb32f&t=d88b7d2e-2609-55f5-9005-b62435c7a771 and https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=29023&view=logs&j=517fe804-fa30-5dc2-1413-330699242c05&t=2e128ad5-2f7f-5333-3f34-c85b8fbc7250 for example.

TomAugspurger · 2020-02-19T22:16:27Z

Seems like these were added in #28777. @jbrockmendel do you have access to a 32-bit OS?

I would think this would be failing on our 32-bit job, but apparently not (or they aren't being run?).

jbrockmendel · 2020-02-19T22:23:00Z

do you have access to a 32-bit OS?

I'll take another try at building on a RasPi

I would think this would be failing on our 32-bit job, but apparently not (or they aren't being run?).

IIRC I un-xfailed this on the theory that the CI would tell us if that was a mistake. I'm surprised this hasn't shown up until now.

TomAugspurger · 2020-02-19T22:28:13Z

Perhaps @toddrme2178 nailed the reason in the original post

values are promotoed to int32 or uint32 targets on i586 the platform (but not on x86_64).

I'm not sure how to check that.

TomAugspurger · 2020-03-02T19:51:57Z

@toddrme2178 are you able to do any more debugging on why these tests are failing? Are you interested in figuring out why our CI is failing to catch these?

Generally speaking, we need help supporting 32-bit systems.

toddrme2178 · 2020-03-04T14:06:51Z

@TomAugspurger I'll take a look when I get a chance.

toddrme2178 · 2020-03-11T19:13:37Z

I have figured out at the very least that the issue only happens in numpy 1.18 and later. Currently only numpy 1.14 is being tested on 32bit. I have expanded the 32bit tests to cover numpy 1.14, 1.15, 1.16, 1.17, 1.18, and master. I don't think these should all be tested normally, but to catch these sorts of issues numpy dev might be tested on 32bit.

That doesn't really explain why the issue is happening, but at least it shows why it wasn't being caught.

np.intc (C int), np.int_ (C long) and np.longlong (C long long) are always distinct type objects, but only two of them are actually different sizes; np.int32 and np.int64 are aliases https://sources.debian.org/src/numpy/1:1.18.4-1/numpy/core/_type_aliases.py/#L110 Author: Rebecca N. Palmer <[email protected]> Bug: pandas-dev/pandas#31856 Forwarded: not-needed pandas-dev/pandas@1660057 Gbp-Pq: Name test_promote_32bit.patch

mroeschke · 2023-11-21T19:07:19Z

Looks like test_maybe_promote_int_with_int is no longer skipped in our CI for our 32 bit build so closing. Can reopen if we see this again

jbrockmendel added the Unreliable Test Unit tests that occasionally fail label Feb 25, 2020

SaturnFromTitan mentioned this issue Feb 27, 2020

TST: refactored test_factorize #32311

Merged

3 tasks

TomAugspurger mentioned this issue Mar 13, 2020

Tests fail on 32bit archs with Python 3.8 and pandas 1.0.2 #32689

Closed

mroeschke added the 32bit 32-bit systems label Apr 19, 2020

simonjayhawkins mentioned this issue Aug 19, 2020

RLS: 1.1.1 #35489

Closed

simonjayhawkins mentioned this issue Nov 27, 2020

remove skip for test_maybe_promote_int_with_int MacPython/pandas-wheels#117

Merged

mroeschke added the Bug label Jul 28, 2021

mroeschke closed this as completed Nov 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`test_maybe_promote_int_with_int` error on 32bit platform #31856

`test_maybe_promote_int_with_int` error on 32bit platform #31856

toddrme2178 commented Feb 10, 2020

TomAugspurger commented Feb 10, 2020

TomAugspurger commented Feb 19, 2020

TomAugspurger commented Feb 19, 2020

jbrockmendel commented Feb 19, 2020

TomAugspurger commented Feb 19, 2020

TomAugspurger commented Mar 2, 2020

toddrme2178 commented Mar 4, 2020

toddrme2178 commented Mar 11, 2020

mroeschke commented Nov 21, 2023

test_maybe_promote_int_with_int error on 32bit platform #31856

test_maybe_promote_int_with_int error on 32bit platform #31856

Comments

toddrme2178 commented Feb 10, 2020

TomAugspurger commented Feb 10, 2020

TomAugspurger commented Feb 19, 2020

TomAugspurger commented Feb 19, 2020

jbrockmendel commented Feb 19, 2020

TomAugspurger commented Feb 19, 2020

TomAugspurger commented Mar 2, 2020

toddrme2178 commented Mar 4, 2020

toddrme2178 commented Mar 11, 2020

mroeschke commented Nov 21, 2023

`test_maybe_promote_int_with_int` error on 32bit platform #31856

`test_maybe_promote_int_with_int` error on 32bit platform #31856