ENH: Infer integer-na in infer_dtype #27392
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gfyoung
added
Dtype Conversions
gfyoung
requested changes
Jul 15, 2019
jiangyue12392
force-pushed
the
integer-na
branch
from
6e252a6 to
f1b9408
Compare
|
@gfyoung Tests added |
jiangyue12392
force-pushed
the
integer-na
branch
from
26a9658 to
514a7c0
Compare
jreback
requested changes
Jul 20, 2019
pandas/_libs/lib.pyx
Outdated
| cdef inline bint is_value_typed(self, object value) except -1: | ||
| return util.is_integer_object(value) or util.is_nan(value) | ||
|
|
||
| cdef inline bint is_array_typed(self) except -1: |
There was a problem hiding this comment.
I am not sure you need this one, is it actually hit?
pandas/_libs/lib.pyx
Outdated
| @@ -1511,6 +1517,21 @@ cpdef bint is_integer_array(ndarray values): | |||
| return validator.validate(values) | |||
|
|
|||
|
|
|||
| cdef class IntegerNaValidator(Validator): | |||
| cdef inline bint is_value_typed(self, object value) except -1: | |||
| return util.is_integer_object(value) or util.is_nan(value) | |||
There was a problem hiding this comment.
I think you need a is_valid_null as well (e.g. see how PeriodValidator is done)
There was a problem hiding this comment.
I changed to util.is_nan(value) and util.is_float_object(value) instead
pandas/core/indexes/base.py
Outdated
| @@ -3196,7 +3197,10 @@ def is_int(v): | |||
| self.get_loc(stop) | |||
| is_positional = False | |||
| except KeyError: | |||
| if self.inferred_type == "mixed-integer-float": | |||
| if ( | |||
There was a problem hiding this comment.
self.inferred_type in [....]
| @@ -577,6 +577,16 @@ def test_integers(self): | |||
| result = lib.infer_dtype(arr, skipna=True) | |||
| assert result == "integer" | |||
|
|
|||
| # GH 27392 | |||
There was a problem hiding this comment.
can you move to inside the definition
| @@ -577,6 +577,16 @@ def test_integers(self): | |||
| result = lib.infer_dtype(arr, skipna=True) | |||
| assert result == "integer" | |||
|
|
|||
| # GH 27392 | |||
| def test_integer_na(self): | |||
There was a problem hiding this comment.
can you parameterize over skipna?
There was a problem hiding this comment.
Parameterization done
jiangyue12392
force-pushed
the
integer-na
branch
2 times, most recently
from
966ac20 to
55cb505
Compare
jreback
requested changes
Jul 23, 2019
doc/source/whatsnew/v0.25.0.rst
Outdated
| @@ -825,6 +825,7 @@ Other API changes | |||
| - :meth:`ExtensionArray.argsort` places NA values at the end of the sorted array. (:issue:`21801`) | |||
| - :meth:`DataFrame.to_hdf` and :meth:`Series.to_hdf` will now raise a ``NotImplementedError`` when saving a :class:`MultiIndex` with extention data types for a ``fixed`` format. (:issue:`7775`) | |||
| - Passing duplicate ``names`` in :meth:`read_csv` will now raise a ``ValueError`` (:issue:`17346`) | |||
| - :meth:`infer_type` will now return integer-na for integer and np.nan mix (:issue:`27283`) | |||
There was a problem hiding this comment.
can you put integer-na in quotes here; move to 1.0
There was a problem hiding this comment.
Updated and moved
jiangyue12392
force-pushed
the
integer-na
branch
from
55cb505 to
fdcaf6e
Compare
added 2 commits
July 24, 2019 14:08
jiangyue12392
force-pushed
the
integer-na
branch
from
93cdef8 to
3f51da7
Compare
jiangyue12392
force-pushed
the
integer-na
branch
from
3f51da7 to
8123014
Compare
jreback
reviewed
Jul 25, 2019
doc/source/whatsnew/v1.0.0.rst
Outdated
| @@ -45,6 +45,7 @@ Backwards incompatible API changes | |||
| Other API changes | |||
| ^^^^^^^^^^^^^^^^^ | |||
|
|
|||
| - :meth:`infer_type` will now return "integer-na" for integer and np.nan mix (:issue:`27283`) | |||
There was a problem hiding this comment.
I think this needs to be :meth:`pandas.api.types.infer_dtype`
jreback
approved these changes
Jul 25, 2019
|
@jreback Green now |
jreback
reviewed
Jul 26, 2019
doc/source/whatsnew/v1.0.0.rst
Outdated
| @@ -45,6 +45,7 @@ Backwards incompatible API changes | |||
| Other API changes | |||
| ^^^^^^^^^^^^^^^^^ | |||
|
|
|||
| - :meth:`pandas.api.types.infer_type` will now return "integer-na" for integer and np.nan mix (:issue:`27283`) | |||
There was a problem hiding this comment.
Sure, updated
|
@gfyoung ok here? |
jiangyue12392
force-pushed
the
integer-na
branch
from
16ce485 to
1c67efb
Compare
gfyoung
reviewed
Jul 28, 2019
doc/source/whatsnew/v1.0.0.rst
Outdated
| @@ -45,6 +45,7 @@ Backwards incompatible API changes | |||
| Other API changes | |||
| ^^^^^^^^^^^^^^^^^ | |||
|
|
|||
| - :meth:`pandas.api.types.infer_dtype` will now return "integer-na" for integer and np.nan mix (:issue:`27283`) | |||
There was a problem hiding this comment.
Double back-ticks around np.nan
gfyoung
reviewed
Jul 28, 2019
jiangyue12392
force-pushed
the
integer-na
branch
from
1c67efb to
f4b6869
Compare
gfyoung
reviewed
Jul 29, 2019
| @@ -577,6 +577,28 @@ def test_integers(self): | |||
| result = lib.infer_dtype(arr, skipna=True) | |||
| assert result == "integer" | |||
|
|
|||
| @pytest.mark.parametrize( | |||
| "arr, skipna, expected", | |||
There was a problem hiding this comment.
You don't need to parameterize expected here.
Notice how it's integer-na when skipa is False and integer when skipna is True.
There was a problem hiding this comment.
Ok, sure, updated
jiangyue12392
force-pushed
the
integer-na
branch
from
f4b6869 to
cfed693
Compare
gfyoung
reviewed
Jul 29, 2019
jiangyue12392
force-pushed
the
integer-na
branch
from
cfed693 to
fad37c8
Compare
|
thanks @jiangyue12392 keep em coming! you are tackling non-trivial issues and making very nice PRs! |
quintusdias
pushed a commit
to quintusdias/pandas_dev
that referenced
this pull request
Aug 16, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diffAs mentioned in #27335, this PR serves as an infrastructure for future changes in the default int type