BUG: pandas.DataFrame.apply gives different results between pandas versions 1.1.4 and 1.2.0

[brandonlind]

• [ x] I have checked that this issue has not already been reported.

• [ x] I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

# Your code here
df2 =pd.DataFrame(dict(one=[str((i/10)*100)+"%" for i in range(5)],
                       two=[str((i/20)*100)+"%" for i in range(10,15)],
                       three=[str((i/30)*100)+"%" for i in range(20,25)]))

# the following two lines give the same result as the other in 1.1.4 but different results in 1.2.0
df2.apply(lambda series: series.str.replace("%","").astype(float)/100, axis=1)  
df2.apply(lambda series: series.str.replace("%","").astype(float)/100, axis=0)

Problem description

[this should explain why the current behaviour is a problem and why the expected output is a better solution]

The two versions of pandas (1.1.4 and 1.2.0) give different results when axis=1

Expected Output

They should be the same when axis=1

Output of pd.show_versions()

even though my "pandas 1.2.0 environment" is python 3.8.5 while my "pandas 1.1.4 environment" is python 3.7.9, the problem is resolved if I revert the python 3.8.5 environment to pandas 1.1.4.

pandas 1.1.4 environment

INSTALLED VERSIONS
------------------
commit           : 67a3d4241ab84419856b84fc3ebc9abcbe66c6b3
python           : 3.7.9.final.0
python-bits      : 64
OS               : Linux
OS-release       : 3.10.0-1160.11.1.el7.x86_64
Version          : #1 SMP Mon Nov 30 13:05:31 EST 2020
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.1.4
numpy            : 1.19.4
pytz             : 2020.4
dateutil         : 2.8.1
pip              : 20.3.3
setuptools       : 51.0.0.post20201207
Cython           : None
pytest           : 6.2.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.6.2
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.2
IPython          : 7.19.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.3.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : 1.5.4
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

pandas 1.2.0 environment

INSTALLED VERSIONS
------------------
commit           : 3e89b4c4b1580aa890023fc550774e63d499da25
python           : 3.8.5.final.0
python-bits      : 64
OS               : Linux
OS-release       : 3.10.0-1160.11.1.el7.x86_64
Version          : #1 SMP Mon Nov 30 13:05:31 EST 2020
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.2.0
numpy            : 1.19.4
pytz             : 2020.5
dateutil         : 2.8.1
pip              : 20.3.3
setuptools       : 51.0.0.post20201207
Cython           : None
pytest           : 6.2.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.6.2
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.2
IPython          : 7.19.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.3.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : 1.5.4
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

[paste the output of pd.show_versions() here leaving a blank line after the details tag]

[rhshadrach]

Thanks for the report - on master I get the result

   one   two     three
0  0.0  0.50  0.666667
1  0.1  0.55  0.700000
2  0.2  0.60  0.733333
3  0.3  0.65  0.766667
4  0.4  0.70  0.800000

This looks reasonable to me. What are you expecting @brandonlind?

[jorisvandenbossche]

Pandas 1.2.0 gives this:

In [2]: df2.apply(lambda series: series.str.replace("%","").astype(float)/100, axis=1)
Out[2]: 
   one  two     three
0  0.0  0.5  0.666667
1  0.0  0.5  0.666667
2  0.0  0.5  0.666667
3  0.0  0.5  0.666667
4  0.0  0.5  0.666667

In [3]: df2.apply(lambda series: series.str.replace("%","").astype(float)/100, axis=0)
Out[3]: 
   one   two     three
0  0.0  0.50  0.666667
1  0.1  0.55  0.700000
2  0.2  0.60  0.733333
3  0.3  0.65  0.766667
4  0.4  0.70  0.800000

so where the first (with axis=1) is wrong.

But it seems this is already fixed in 1.2.1 (@brandonlind pandas 1.2.1 was released 2 days ago, so if you update pandas it should work again)

Might still be worth adding a test for it (don't know if the patch that fixed it was for something similar).

[simonjayhawkins]

duplicate of #38979, fixed (and tests added) in #39188

[brandonlind]

thanks everyone!

[rhshadrach]

Thanks @jorisvandenbossche and @simonjayhawkins