BUG: Series.str.strip() removes numeric and bool values instead of only string whitespace

[mspacek]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np
s = pd.Series(['hello ', ' goodbye', 1.0, 0.0, 1, 0, True, False, np.nan])
>>> s
0      hello 
1     goodbye
2         1.0
3         0.0
4           1
5           0
6        True
7       False
8         NaN
dtype: object

>>> s.values
array(['hello ', ' goodbye', 1.0, 0.0, 1, 0, True, False, nan],
      dtype=object)

>>> s.str.strip()
0      hello
1    goodbye
2        NaN
3        NaN
4        NaN
5        NaN
6        NaN
7        NaN
8        NaN
dtype: object

>>> s.str.strip().values
array(['hello', 'goodbye', nan, nan, nan, nan, nan, nan, nan],
      dtype=object)

Issue Description

The documentation for Series.str.strip() says:

Strip whitespaces (including newlines) or a set of specified characters from each string in the Series/Index from left and right sides. Equivalent to str.strip().

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.strip.html

I assumed that for Series with mixed data types, i.e. that have dtype=object, this method would only strip whitespace from string-like entries. Instead, to my surprise, it also replaces numeric and bool values with NaN.

I assume this isn't intentional, but if it is intentional, it should be documented.

Expected Behavior

Numeric and bool values should be left untouched. Series.str.strip() should only apply to string values. Here's my naive implementation:

def strip_df(df, colnames=None):
    """Strip leading and trailing whitespace from string-like values of `colnames`
    in DataFrame `df`"""
    df = df.copy() # don't modify in-place
    if colnames is None:
        colnames = df.columns # all columns
    else:
        colnames = np.intersect1d(colnames, df.columns)
    for colname in colnames:
        for i, val in df[colname].iteritems():
            try:
                df.loc[i, colname] = val.strip()
            except AttributeError: # not a string-like value
                pass
    return df

Installed Versions

INSTALLED VERSIONS

commit : 66e3805
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-91-lowlatency
Version : #102-Ubuntu SMP PREEMPT Fri Nov 5 18:18:39 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.5
numpy : 1.21.4
pytz : 2021.3
dateutil : 2.7.3
pip : 21.3.1
setuptools : 45.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.5.0
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.30.1
pandas_datareader: None
bs4 : 4.8.2
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.5.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[simonjayhawkins]

Thanks @mspacek for the report.

I assume this isn't intentional, but if it is intentional, it should be documented.

This is not a bug, it is longstanding, tested and expected behavior which is unlikely to be changed.

PR to clarify the behavior for mixed type object Series welcome.

from https://pandas.pydata.org/pandas-docs/dev/user_guide/text.html#string-methods

Generally speaking, the .str accessor is intended to work only on strings. With very few exceptions, other uses are not supported, and may be disabled at a later point.

and there is a discussion issue open to consider deprecating the str accessor on object dtype Series #29710.

and tbc it is not just .str.strip() but also other str methods e.g. .str.upper()

Numeric and bool values should be left untouched. Series.str.strip() should only apply to string values. Here's my naive implementation:

as a workaround for the specific use case to get the "expected" behavior, could maybe consider something like...

s = pd.Series(["hello ", " goodbye", 1.0, 0.0, 1, 0, True, False, np.nan])
s[s.str.strip().notnull()] = s.str.strip()
print(s)

0      hello
1    goodbye
2          1
3          0
4          1
5          0
6       True
7      False
8        NaN
dtype: object

[mspacek]

Thanks for the reply @simonjayhawkins ! I guess I find the behaviour very surprising, since it leads to silent data loss, which to me qualifies as a major bug. Good to see though that it's being considered for deprecation, and that the documentation is being updated.

[lwardrope]

Hi,
I'm a first time contributor. Is this issue still open, I would like to contribute to this?

[pree-T]

May I work on this?

[Anunaya07]

Hello I am first time contributor. If the issue is still open I would like to work on it?

[ChrisAlbertsen]

take