BUG: Apply pd.get_dummies() on string type columns of pandas dataframe do nothing?

[stevesolun]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

I have the following dataframe:

df = pd.DataFrame({'a': [6.6, -5.2, 2.1, 3.3, 1.1],
              'b': ['a', 'a', 'c', 'b', 'a'],
              'c': ['kfr', 'kfr', 'lu', 'ku', 'lu'],
              'd': ['t', 's', 's', 't', 'a']})

All the dtypes (columns b, c, d) are of type string.

If I call df = df.convert_dtypes(), and then call pd.get_dummies(), nothing happens:

df = pd.DataFrame({'a': [6.6, -5.2, 2.1, 3.3, 1.1],
              'b': ['a', 'a', 'c', 'b', 'a'],
              'c': ['kfr', 'kfr', 'lu', 'ku', 'lu'],
              'd': ['t', 's', 's', 't', 'a']})
df = df.convert_dtypes()
pd.get_dummies(df)

But if I will change the df.convert_dtypes() to df.convert_dtypes(convert_string=False) it will work as expected.
Why is this happening? Is it a bug?

Expected Behavior

pandas pd.get_dummies() should work on strings columns dtype also.

Installed Versions

1.3.5

[cheruiyot-amon]

when you write the pd.get_dummies() without the dataframe inside the bracket, it will throw some errors. The best you can do is create a variable say ,,"dummies"
dummies=pd.get_dummies(name_of_dataframe)

[stevesolun]

when you write the pd.get_dummies() without the dataframe inside the bracket, it will throw some errors. The best you can do is create a variable say ,,"dummies" dummies=pd.get_dummies(name_of_dataframe)

What does it do to fix the bug?

[cheruiyot-amon]

It will create dummies for data series in a dataframe. It will convert the strings into zeros and ones .

…

On Tue, Dec 21, 2021, 12:37 PM Steve Solun ***@***.***> wrote: when you write the pd.get_dummies() without the dataframe inside the bracket, it will throw some errors. The best you can do is create a variable say ,,"dummies" dummies=pd.get_dummies(name_of_dataframe) What does it do to fix the bug? — Reply to this email directly, view it on GitHub <#44965 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMO3FYBLB6FHNK7MYILSQ5LUSBDFNANCNFSM5KLF6V6A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[cheruiyot-amon]

Just write this code:
dummies=pd.get_dummies(df)
print(dummies)
.This will fix your problem

[stevesolun]

I am sorry but it's not the problem here...
The problem is that if I am using convert_dtypes() on my df before get_dummies() it will return the same df without any change. Please read the question again and go step by step. Your solution doesn't solve the bug.

[cheruiyot-amon]

For your case, I think it is not possible to convert strings to integers, float, or even boolean. However, you can convert an integer to float and vice versa.

[stevesolun]

Did you run my example?
I am able to solve it if I will explicitly tell convert_dtypes() to skip converting strings.

[cheruiyot-amon]

Yes. I did run your example. could you please send a screenshot a code so that i could have a look at it

[stevesolun]

I am sorry but if you were able to reproduce the issue, can you please suggest a fix? You are one of the pandas developers?
Maybe we can create a Zoom call?

[cheruiyot-amon]

I am aspiring to become one. I started learning data science recently.

…

On Wed, Dec 22, 2021, 5:46 PM Steve Solun ***@***.***> wrote: I am sorry but if you were able to reproduce the issue, can you please suggest a fix? You are one of the pandas developers? Maybe we can create a Zoom call? — Reply to this email directly, view it on GitHub <#44965 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMO3FYASRMR2AZTTNT34T5TUSHQETANCNFSM5KLF6V6A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[MarcoGorelli]

Hi @stevesolun

I think @cheruiyot-amon 's point is that your example doesn't run as it's written - in order to expedite resolution, could you fix it up so it can be easily copy-and-pasted please?

[stevesolun]

@MarcoGorelli sure. Done

[asishm]

To minimize this

>>> df = pd.DataFrame({'a': ['a', 'b']})
>>> print(pd.get_dummies(df))
   a_a  a_b
0    1    0
1    0    1
>>> print(pd.get_dummies(df.convert_dtypes()))
   a
0  a
1  b

[MarcoGorelli]

I see - thanks for the report, and @asishm for the minimal code !

[asishm]

Issue is because pd.get_dummies only selects the dtypes - object, category to encode (when passed a DataFrame), but with convert_dtypes they are now of dtype string. However, with a Series, it factorizes regardless of the dtype.

pandas/pandas/core/reshape/reshape.py

Line 913 in a3c0e7b

dtypes_to_encode = ["object", "category"]

[stevesolun]

@asishm @lithomas1 thanks a lot!
It's a bug or by design? Does it mean that the dev team will change the behavior of get_dummies to support strings also?

[lukemanley]

take