BUG: float-like string, trailing 0 truncation #38759

mzeitlin11 · 2020-12-28T21:01:04Z

closes REGR: repr of stringified floats (e.g. "3.50") drop ending "0" #38708
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

rhshadrach

Changes look good to me, just some minor test/doc requests.

rhshadrach · 2020-12-28T21:13:30Z

doc/source/whatsnew/v1.2.1.rst

@@ -15,6 +15,7 @@ including other versions of pandas.
 Fixed regressions
 ~~~~~~~~~~~~~~~~~
 - The deprecated attributes ``_AXIS_NAMES`` and ``_AXIS_NUMBERS`` of :class:`DataFrame` and :class:`Series` will no longer show up in ``dir`` or ``inspect.getmembers`` calls (:issue:`38740`)
+- Bug in float-like strings having trailing 0's truncated (:issue:`38708`)


Can you add in "repr of" and "after the decimal" here for clarity that it is just a bug in regards to presentation (which is still important!).

these are not floats, rather strings in an object dtype

rhshadrach · 2020-12-28T21:35:20Z

pandas/tests/series/test_repr.py

@@ -237,6 +237,18 @@ def test_series_repr_nat(self):
        )
        assert result == expected

+    def test_series_repr_float_like_object_no_truncate(self):


Can you parametrize/combine these two tests using the parameters data and expected, and add in cases where np.nan/None values are present, including data = [np.nan] and data = [None].

jreback · 2020-12-28T21:45:49Z

doc/source/whatsnew/v1.2.1.rst

@@ -15,6 +15,7 @@ including other versions of pandas.
 Fixed regressions
 ~~~~~~~~~~~~~~~~~
 - The deprecated attributes ``_AXIS_NAMES`` and ``_AXIS_NUMBERS`` of :class:`DataFrame` and :class:`Series` will no longer show up in ``dir`` or ``inspect.getmembers`` calls (:issue:`38740`)
+- Bug in float-like strings having trailing 0's truncated (:issue:`38708`)


these are not floats, rather strings in an object dtype

jreback · 2020-12-28T21:46:31Z

pandas/io/formats/format.py

@@ -1310,7 +1310,9 @@ def _format(x):
                    tpl = " {v}"
                fmt_values.append(tpl.format(v=_format(v)))

-        fmt_values = _trim_zeros_float(str_floats=fmt_values, decimal=".")
+        fmt_values = _trim_zeros_float(


there is a larger issue to address first. these are not floats at all. these are object dtypes that are reprering. why are they hitting this path is the issue. we likley do not want to touch the formatters.

there are likley getting inferred as floats and that is why the formatter is picked. this is incorrect.

This is the GenericArrayFormatter - seems reasonable for object type data. I think hitting _trim_zeros_float in GenericArrayFormatter makes sense because of a case like

s = pd.Series([1.20, "1.00"]) repr(s)

which should hit _trim_zeros_float so that 1.20 is truncated to 1.2.

ok, sure but a string should not be truncated. that's where the issue is.

i see the issue here.

instead of _trim_zeros_float being a function that accepts a ndarray/list of strings, just have it process a single value, then call it when needed in the loop on L1299.

this makes the code much simpler. i am not entirely sure why it was done this way.

I think the reason why it was done that way is that floats are formatted with fixed-width, e.g.

s = pd.Series([1.230, 1.20000]) repr(s)

gives

0 1.23 1 1.20 dtype: float64

_trim_zeros_float was designed for usage in FloatArrayFormatter (so it needs the whole list to keep fixed-width, since it only truncates if all values can be truncated). The usage of this function in GenericArrayFormatter is what caused the regression. I think then it makes sense to keep _trim_zeros_float as is for that original purpose, and GenericArrayFormatter can use a much simpler solution for truncation since it does not have the same fixed width requirements.

ok that sounds fine, tests are pretty sensitive here so you can refactor.

Thanks for the help here, this is much cleaner!

jreback · 2020-12-29T00:12:30Z

pandas/tests/series/test_repr.py

+        ],
+    )
+    def test_repr_str_float_truncation(self, data, expected):
+        series = Series(data)


can you add the issue number here

jreback · 2020-12-29T00:13:38Z

pandas/tests/series/test_repr.py

@@ -237,6 +237,24 @@ def test_series_repr_nat(self):
        )
        assert result == expected

+    @pytest.mark.parametrize(


tests go in pandas/tests/io/formats/test_format.py (as that is where all of the formatting tests are)

put after this test test_float_trim_zeros

jreback · 2020-12-29T00:14:56Z

also pls rebase

mzeitlin11 · 2020-12-29T00:30:29Z

Moved tests, added issue number, fixed conflict

jreback · 2020-12-29T03:18:57Z

thanks @mzeitlin11

jreback · 2020-12-29T03:19:20Z

@meeseeksdev backport 1.2.x

…cation

…8769) Co-authored-by: mzeitlin11 <[email protected]>

* BUG: float-like string, trailing 0 truncation * Don't use _trim_zeros_float

BUG: float-like string, trailing 0 truncation

4817541

mzeitlin11 added Regression Functionality that used to work in a prior pandas version Output-Formatting __repr__ of pandas objects, to_string labels Dec 28, 2020

Fix merge conflict

26d21f4

rhshadrach requested changes Dec 28, 2020

View reviewed changes

jreback requested changes Dec 28, 2020

View reviewed changes

Don't use _trim_zeros_float

d1d2be1

jreback requested changes Dec 29, 2020

View reviewed changes

jreback added this to the 1.2.1 milestone Dec 29, 2020

Move test, add issue number

098012f

jreback approved these changes Dec 29, 2020

View reviewed changes

jreback merged commit 7f912a4 into pandas-dev:master Dec 29, 2020

meeseeksmachine mentioned this pull request Dec 29, 2020

Backport PR #38759 on branch 1.2.x (BUG: float-like string, trailing 0 truncation) #38769

Merged

meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Dec 29, 2020

Backport PR pandas-dev#38759: BUG: float-like string, trailing 0 trun…

6da30a2

…cation

mzeitlin11 deleted the bug/float_like_str_repr branch December 29, 2020 03:20

gfyoung pushed a commit that referenced this pull request Dec 29, 2020

Backport PR #38759: BUG: float-like string, trailing 0 truncation (#3…

c348cbe

…8769) Co-authored-by: mzeitlin11 <[email protected]>

luckyvs1 pushed a commit to luckyvs1/pandas that referenced this pull request Jan 20, 2021

BUG: float-like string, trailing 0 truncation (pandas-dev#38759)

7f27560

* BUG: float-like string, trailing 0 truncation * Don't use _trim_zeros_float

This was referenced Apr 9, 2021

REGR: object column repr not respecting float format #40850

Merged

BUG: Calling to_html with float_format strips all trailing zeros if an integer string is returned from the formatter #40024

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: float-like string, trailing 0 truncation #38759

BUG: float-like string, trailing 0 truncation #38759

mzeitlin11 commented Dec 28, 2020

rhshadrach left a comment

rhshadrach Dec 28, 2020

jreback Dec 28, 2020

mzeitlin11 Dec 28, 2020

rhshadrach Dec 28, 2020

mzeitlin11 Dec 28, 2020

jreback Dec 28, 2020

jreback Dec 28, 2020

jreback Dec 28, 2020

mzeitlin11 Dec 28, 2020

jreback Dec 28, 2020

jreback Dec 28, 2020

mzeitlin11 Dec 28, 2020 •

edited

Loading

jreback Dec 28, 2020

mzeitlin11 Dec 28, 2020

jreback Dec 29, 2020

jreback Dec 29, 2020

jreback Dec 29, 2020

jreback commented Dec 29, 2020

mzeitlin11 commented Dec 29, 2020

jreback commented Dec 29, 2020

jreback commented Dec 29, 2020

BUG: float-like string, trailing 0 truncation #38759

BUG: float-like string, trailing 0 truncation #38759

Conversation

mzeitlin11 commented Dec 28, 2020

rhshadrach left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzeitlin11 Dec 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Dec 29, 2020

mzeitlin11 commented Dec 29, 2020

jreback commented Dec 29, 2020

jreback commented Dec 29, 2020

mzeitlin11 Dec 28, 2020 •

edited

Loading