Skip to content

BUG/CLN: Clean float / complex string formatting #36799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Oct 14, 2020
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
406ed4d
CLN: Clean float / complex string formatting
dsaxton Oct 2, 2020
a1228c9
Fix
dsaxton Oct 2, 2020
eabef62
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 2, 2020
72be97f
Fix and test
dsaxton Oct 2, 2020
e1d1ba3
Fix
dsaxton Oct 2, 2020
d38839c
Fix
dsaxton Oct 2, 2020
0da1c46
Nit
dsaxton Oct 2, 2020
4ea7dcf
Nit
dsaxton Oct 2, 2020
376b06e
Change doc
dsaxton Oct 2, 2020
473d674
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 2, 2020
e564fee
Escape
dsaxton Oct 2, 2020
6cec317
Add failing test
dsaxton Oct 2, 2020
50ccbc0
Edit
dsaxton Oct 2, 2020
e725cb2
Maybe
dsaxton Oct 2, 2020
3a94daa
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 2, 2020
03e8cab
Comment
dsaxton Oct 2, 2020
46949fd
Remove
dsaxton Oct 2, 2020
11f20d0
Compile and comment
dsaxton Oct 3, 2020
44f0792
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 3, 2020
229b5fc
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 3, 2020
eec9d89
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 8, 2020
0e0dd92
Move into other function
dsaxton Oct 8, 2020
956ecf3
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 8, 2020
25dd8f1
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 10, 2020
eca0413
Note
dsaxton Oct 11, 2020
00b71b2
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 11, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions pandas/core/frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -2650,11 +2650,11 @@ def memory_usage(self, index=True, deep=False) -> Series:
>>> df = pd.DataFrame(data)
>>> df.head()
int64 float64 complex128 object bool
0 1 1.0 1.000000+0.000000j 1 True
1 1 1.0 1.000000+0.000000j 1 True
2 1 1.0 1.000000+0.000000j 1 True
3 1 1.0 1.000000+0.000000j 1 True
4 1 1.0 1.000000+0.000000j 1 True
0 1 1.0 1.0+0.0j 1 True
1 1 1.0 1.0+0.0j 1 True
2 1 1.0 1.0+0.0j 1 True
3 1 1.0 1.0+0.0j 1 True
4 1 1.0 1.0+0.0j 1 True

>>> df.memory_usage()
Index 128
Expand Down
33 changes: 25 additions & 8 deletions pandas/io/formats/format.py
Original file line number Diff line number Diff line change
Expand Up @@ -1473,9 +1473,10 @@ def format_values_with(float_format):

if self.fixed_width:
if is_complex:
result = _trim_zeros_complex(values, self.decimal, na_rep)
result = _trim_zeros_complex(values, self.decimal)
result = _post_process_complex(result)
else:
result = _trim_zeros_float(values, self.decimal, na_rep)
result = _trim_zeros_float(values, self.decimal)
return np.asarray(result, dtype="object")

return values
Expand Down Expand Up @@ -1855,29 +1856,45 @@ def just(x):
return result


def _trim_zeros_complex(
str_complexes: np.ndarray, decimal: str = ".", na_rep: str = "NaN"
) -> List[str]:
def _trim_zeros_complex(str_complexes: np.ndarray, decimal: str = ".") -> List[str]:
"""
Separates the real and imaginary parts from the complex number, and
executes the _trim_zeros_float method on each of those.
"""
return [
"".join(_trim_zeros_float(re.split(r"([j+-])", x), decimal, na_rep))
"".join(_trim_zeros_float(re.split(r"([j+-])", x), decimal))
for x in str_complexes
]


def _post_process_complex(complex_strings: List[str]) -> List[str]:
"""
Zero pad complex number strings produced by _trim_zeros_complex.
"""
lengths = [len(s) for s in complex_strings]
max_length = max(lengths)
padded = [
s[: -((k - 1) // 2 + 1)] # real part
+ (max_length - k) // 2 * "0"
+ s[-((k - 1) // 2 + 1) : -((k - 1) // 2)] # + / -
+ s[-((k - 1) // 2) : -1] # imaginary part
+ (max_length - k) // 2 * "0"
+ s[-1]
for s, k in zip(complex_strings, lengths)
]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be safer to split real and imaginary parts via +- and then process decimal and fractional parts by splitting via the dot? This way you would not need to rely on the symmetry of the original string provided.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean trim zeros after splitting into fractional non-fractional parts? I think the trimming has to be done with the decimal there. (I realize this helper is very confusing, and there's likely a better way to do this.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I mean trimming zeros after splitting into fractional and non-fractional parts. Since a dot char would always split float number, there is no risk to introduce a bug IMHO (even if there is no dot at all).

Copy link
Member Author

@dsaxton dsaxton Oct 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think then you have to keep track of which parts are fractional and then only trim those?

However this part is not doing any actual trimming, it's correcting for the fact that the previous function is now trimming "too much." (It trims the real and imaginary parts of each complex number independently, so they aren't aligned afterwards. Rather than rewrite the other function I found it easier to do this post-processing.)

return padded


def _trim_zeros_float(
str_floats: Union[np.ndarray, List[str]], decimal: str = ".", na_rep: str = "NaN"
str_floats: Union[np.ndarray, List[str]], decimal: str = "."
) -> List[str]:
"""
Trims zeros, leaving just one before the decimal points if need be.
"""
trimmed = str_floats

def _is_number(x):
return x != na_rep and not x.endswith("inf")
return re.match(fr"\s*-?[0-9]+(\{decimal}[0-9]*)?", x) is not None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you compile this and put it on the class / variable


def _cond(values):
finite = [x for x in values if _is_number(x)]
Expand Down
7 changes: 7 additions & 0 deletions pandas/tests/io/formats/test_format.py
Original file line number Diff line number Diff line change
Expand Up @@ -3432,3 +3432,10 @@ def test_format_remove_leading_space_dataframe(input_array, expected):
# GH: 24980
df = pd.DataFrame(input_array).to_string(index=False)
assert df == expected


def test_to_string_complex_number_trims_zeros():
s = pd.Series([1.000000 + 1.000000j, 1.0 + 1.0j, 1.05 + 1.0j])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why should these have 2 decimal zeros and not 1 likely ordinary floats? Where did you get the expected output from?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a 1.05 in the last element, the expected output is similar to what happens for floats:

[ins] In [2]: s = pd.Series([1.0, 1.0000, 1.05])

[ins] In [3]: s
Out[3]: 
0    1.00
1    1.00
2    1.05
dtype: float64

result = s.to_string()
expected = "0 1.00+1.00j\n1 1.00+1.00j\n2 1.05+1.00j"
assert result == expected