String dtype: still return nullable NA-variant in object inference (maybe_converts_object) if requested
#59487
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Comment on lines
-2702
to
2712
| if using_string_dtype() and is_string_array(objects, skipna=True): | ||
| if convert_to_nullable_dtype and is_string_array(objects, skipna=True): | ||
| from pandas.core.arrays.string_ import StringDtype | ||
|
|
||
| dtype = StringDtype(na_value=np.nan) | ||
| dtype = StringDtype() | ||
| return dtype.construct_array_type()._from_sequence(objects, dtype=dtype) | ||
|
|
||
| elif convert_to_nullable_dtype and is_string_array(objects, skipna=True): | ||
| elif using_string_dtype() and is_string_array(objects, skipna=True): | ||
| from pandas.core.arrays.string_ import StringDtype | ||
|
|
||
| dtype = StringDtype() | ||
| dtype = StringDtype(na_value=np.nan) | ||
| return dtype.construct_array_type()._from_sequence(objects, dtype=dtype) |
There was a problem hiding this comment.
Essentially this diff is just switching the order of the if/elif blocks, to first check if we want a nullable dtype and only then check if we want the future default string dtype.
pandas/core/arrays/datetimelike.py
Outdated
Comment on lines
672
to
673
| value = sanitize_array(value, index=None) | ||
| value = ensure_wrapped_if_datetimelike(value) |
There was a problem hiding this comment.
This is essentially the default input sanitation/inference we do for the Series constructor (and other places), so switched to use that here instead of pd.array(..) (which doesn't use the default dtypes for all types)
(this mostly impacts the name of the type in the error message)
jorisvandenbossche
changed the title
maybe_converts_object) if requested
I think generally should just stick with |
mroeschke
approved these changes
Aug 21, 2024
|
Thanks @jorisvandenbossche |
lithomas1
pushed a commit
to lithomas1/pandas
that referenced
this pull request
Sep 8, 2024
…maybe_converts_object`) if requested (pandas-dev#59487) * String dtype: maybe_converts_object give precedence to nullable dtype * update datetimelike input validation * update tests and remove xfails * explicitly test pd.array() behaviour (remove xfail) * fixup allow_2d * undo changes related to datetimelike input validation * fix test for str on current main --------- Co-authored-by: Matthew Roeschke <[email protected]> (cherry picked from commit 851639d)
jorisvandenbossche
added a commit
to WillAyd/pandas
that referenced
this pull request
Sep 20, 2024
…maybe_converts_object`) if requested (pandas-dev#59487) * String dtype: maybe_converts_object give precedence to nullable dtype * update datetimelike input validation * update tests and remove xfails * explicitly test pd.array() behaviour (remove xfail) * fixup allow_2d * undo changes related to datetimelike input validation * fix test for str on current main --------- Co-authored-by: Matthew Roeschke <[email protected]>
jorisvandenbossche
added a commit
to WillAyd/pandas
that referenced
this pull request
Oct 2, 2024
…maybe_converts_object`) if requested (pandas-dev#59487) * String dtype: maybe_converts_object give precedence to nullable dtype * update datetimelike input validation * update tests and remove xfails * explicitly test pd.array() behaviour (remove xfail) * fixup allow_2d * undo changes related to datetimelike input validation * fix test for str on current main --------- Co-authored-by: Matthew Roeschke <[email protected]>
jorisvandenbossche
added a commit
to WillAyd/pandas
that referenced
this pull request
Oct 2, 2024
…maybe_converts_object`) if requested (pandas-dev#59487) * String dtype: maybe_converts_object give precedence to nullable dtype * update datetimelike input validation * update tests and remove xfails * explicitly test pd.array() behaviour (remove xfail) * fixup allow_2d * undo changes related to datetimelike input validation * fix test for str on current main --------- Co-authored-by: Matthew Roeschke <[email protected]>
jorisvandenbossche
added a commit
to WillAyd/pandas
that referenced
this pull request
Oct 3, 2024
…maybe_converts_object`) if requested (pandas-dev#59487) * String dtype: maybe_converts_object give precedence to nullable dtype * update datetimelike input validation * update tests and remove xfails * explicitly test pd.array() behaviour (remove xfail) * fixup allow_2d * undo changes related to datetimelike input validation * fix test for str on current main --------- Co-authored-by: Matthew Roeschke <[email protected]>
jorisvandenbossche
added a commit
to WillAyd/pandas
that referenced
this pull request
Oct 7, 2024
…maybe_converts_object`) if requested (pandas-dev#59487) * String dtype: maybe_converts_object give precedence to nullable dtype * update datetimelike input validation * update tests and remove xfails * explicitly test pd.array() behaviour (remove xfail) * fixup allow_2d * undo changes related to datetimelike input validation * fix test for str on current main --------- Co-authored-by: Matthew Roeschke <[email protected]>
jorisvandenbossche
added a commit
that referenced
this pull request
Oct 9, 2024
…maybe_converts_object`) if requested (#59487) * String dtype: maybe_converts_object give precedence to nullable dtype * update datetimelike input validation * update tests and remove xfails * explicitly test pd.array() behaviour (remove xfail) * fixup allow_2d * undo changes related to datetimelike input validation * fix test for str on current main --------- Co-authored-by: Matthew Roeschke <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
lib.maybe_converts_object()function (which is used for object dtype inference in various places) has aconvert_to_nullable_dtype=Flase/Truekeyword to control whether to return dtypes using NA or not. Currently we were giving the futureinfer_stringoption priority. But also when the future dtype is enabled, we should still return an NA-variant of the dtype if explicitly asked for. So this PR switches the order of those two checks to let theconvert_to_nullable_dtypekeyword take precedence over the option.This came up in SQL tests, but actually this also impacts the
pd.array(..)behaviour. Currently on main,pd.array(["some", "strings"])will infer the nullableStringDtype(), consistent with the other dtypes that have such a nullable variant (i.e. returning our Int64 instead of np.int64 for integers, etc).This is tested behaviour, but we were still xfailing those tests when
future.infer_stringwas enabled. With the change in this PR, also in the futurepd.array(..)will infer the NA-variant.(sidenote, we should update this simple
convert_to_nullable_dtypeTrue/False flag to something that allows passing whether we want a numpy_nullable or pyarrow array, but that's for a separate issue)