TYP: prep _generate_range_overflow_safe for numpy 1.20

[simonjayhawkins]

difficult call, will replace

pandas/core/arrays/_ranges.py:153: error: Incompatible return value type (got "signedinteger[_64Bit]", expected "int") [return-value]
pandas/core/arrays/_ranges.py:171: error: Incompatible return value type (got "unsignedinteger[_64Bit]", expected "int") [return-value]

with

pandas/core/arrays/_ranges.py:129: error: Argument 1 to "_generate_range_overflow_safe" has incompatible type "Union[int, number[Any]]"; expected "Union[int, integer[Any]]" [arg-type]
pandas/core/arrays/_ranges.py:129: error: Argument 2 to "_generate_range_overflow_safe" has incompatible type "Union[int, number[Any]]"; expected "Union[int, integer[Any]]" [arg-type]
pandas/core/arrays/_ranges.py:137: error: Argument 2 to "_generate_range_overflow_safe" has incompatible type "Union[int, number[Any]]"; expected "Union[int, integer[Any]]" [arg-type]
pandas/core/arrays/_ranges.py:138: error: Argument 2 to "_generate_range_overflow_safe" has incompatible type "Union[int, number[Any]]"; expected "Union[int, integer[Any]]" [arg-type]

these are from arithmetic operations on np.integer (including floordiv) which AFAICT should return np.integer and not np.number

can't add the ignores yet, since we will get unused ignore mypy messages - active branch with fixes https://github.com/pandas-dev/pandas/compare/master...simonjayhawkins:numpy-fixes?expand=1

[jreback]

we shouldn't do this specifically here, almost anywhere we accept int we also accept np.integer, e.g. thisi by using is_integer checking.

[simonjayhawkins]

yeah, we should use an alias in pandas._typing, however, my concern is putting that in place before we transition to mypy checking using numpy types is that there will be many more mypy errors that won't yet be reported/visible, potentially making the transition harder.

see also scipy/scipy#10844 (review) and numpy/numpy#18096

Here, was just using the Union where we internally explicitly pass a np.int to a function, for now.

I'm not convinced we should do this yet, either. so can use this PR for discussion.

also need to convince myself, that those new errors are actually numpy issues with the return types of np.integer before we use _int = Union[int, np.integer].

import numpy as np

py_int = 42
reveal_type(py_int)
reveal_type(py_int // 2)
reveal_type(py_int - 1)

np_int = np.int64(np.iinfo(np.int64).min)
reveal_type(np_int)
reveal_type(np_int // 2)
reveal_type(np_int - 1)
print(np_int - 1)

np.integer(np_int)

np_integer: np.integer
reveal_type(np_integer)
reveal_type(np_integer // 2)
reveal_type(np_integer - 1)

test.py:4: note: Revealed type is 'builtins.int'
test.py:5: note: Revealed type is 'builtins.int'
test.py:6: note: Revealed type is 'builtins.int'
test.py:9: note: Revealed type is 'numpy.signedinteger[numpy.typing._64Bit*]'
test.py:10: note: Revealed type is 'numpy.signedinteger[Any]'
test.py:11: note: Revealed type is 'numpy.signedinteger[Any]'
test.py:14: error: Cannot instantiate abstract class 'integer' with abstract attribute '__init__'  [abstract]
test.py:17: note: Revealed type is 'numpy.integer[Any]'
test.py:18: note: Revealed type is 'numpy.number[Any]'
test.py:19: note: Revealed type is 'numpy.number[Any]'

[jreback]

yeah, we should use an alias in pandas._typing, however, my concern is putting that in place before we transition to mypy checking using numpy types is that there will be many more mypy errors that won't yet be reported/visible, potentially making the transition harder.

how does not using an alias help?

[simonjayhawkins]

I guess while i keep #36092 up to date, we do have some visibility.

but if we start using an alias now, while numpy types resolve to Any we may increase the number of mypy errors to resolve when transitioning to using numpy types.

[jreback]

so would it be better then to not type this output? avoiding both scenarios?

[simonjayhawkins]

indeed removing some of the problematic type annotations could be a better short term solution, and add them back once we have numpy types and we have mypy errors visible to help resolve.

[jreback]

kk great. yeah certainly love to have more annotations, but if they are going to be false positives then we shouldn't add (now)

[simonjayhawkins]

closing while we discuss #39513