Skip to content

BUG: fix infer_dtype for StringDtype #31877

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Feb 12, 2020
Merged

BUG: fix infer_dtype for StringDtype #31877

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a20f41c
BUG: fix infer_dtype for StringDtype
jorisvandenbossche Feb 11, 2020
fcaddd3
fix quoting
jorisvandenbossche Feb 11, 2020
a928d0b
Merge remote-tracking branch 'upstream/master' into infer-string
jorisvandenbossche Feb 12, 2020
f04932c
add list case to fixture
jorisvandenbossche Feb 12, 2020
92b0083
parametrize class, skipna, data
jorisvandenbossche Feb 12, 2020
1d7a1d3
Merge remote-tracking branch 'upstream/master' into infer-string
jorisvandenbossche Feb 12, 2020
45947a8
add null
jorisvandenbossche Feb 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions doc/source/whatsnew/v1.0.2.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ Bug fixes

- Using ``pd.NA`` with :meth:`DataFrame.to_json` now correctly outputs a null value instead of an empty object (:issue:`31615`)


**Experimental dtypes**

- Fix bug in :meth:`DataFrame.convert_dtypes` for columns that were already using the ``"string"`` dtype (:issue:`31731`).

.. ---------------------------------------------------------------------------

.. _whatsnew_102.contributors:
Expand Down
2 changes: 1 addition & 1 deletion pandas/_libs/lib.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1005,7 +1005,7 @@ _TYPE_MAP = {
'complex64': 'complex',
'complex128': 'complex',
'c': 'complex',
'string': 'bytes',
'string': 'string',
'S': 'bytes',
'U': 'string',
'bool': 'boolean',
Expand Down
14 changes: 14 additions & 0 deletions pandas/tests/dtypes/test_inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1200,6 +1200,20 @@ def test_interval(self):
inferred = lib.infer_dtype(pd.Series(idx), skipna=False)
assert inferred == "interval"

def test_string_dtype(self):
# StringArray
arr = pd.array(["a", "b", pd.NA], dtype="string")
for val in [list(arr), arr, pd.Series(arr)]:
inferred = lib.infer_dtype(val)
assert inferred == "string"

def test_boolean_dtype(self):
# BooleanArray
arr = pd.array([True, False, pd.NA], dtype="boolean")
for val in [list(arr), arr, pd.Series(arr)]:
inferred = lib.infer_dtype(val)
assert inferred == "boolean"


class TestNumberScalar:
def test_is_number(self):
Expand Down
7 changes: 7 additions & 0 deletions pandas/tests/series/methods/test_convert_dtypes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,3 +246,10 @@ def test_convert_dtypes(self, data, maindtype, params, answerdict):

# Make sure original not changed
tm.assert_series_equal(series, copy)

def test_convert_string_dtype(self):
# https://github.com/pandas-dev/pandas/issues/31731 -> converting columns
# that are already string dtype
df = pd.DataFrame({"A": ["a", "b", "c"], "B": ["ä", "ö", "ü"]}, dtype="string")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you parameterize using the nulls_fixture (and no null as you have now) as well

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may want to have a null_fixture_for_strings e.g. [None, np.nan, pd.NA]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need to parametrize for nulls here, I think. I am creating this with dtype string, and once you have that dtype, there is no difference for different nulls. So it would be testing construction with different nulls, which isn't the goal here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are there other tests that convert_dtypes on strings with nulls?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

convert_dtypes doesn't care about nulls or not if the dtype is already "string".

But added one NA in the values.

result = df.convert_dtypes()
tm.assert_frame_equal(df, result)