Skip to content

BUG/CLN: Clean float / complex string formatting #36799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Oct 14, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
406ed4d
CLN: Clean float / complex string formatting
dsaxton Oct 2, 2020
a1228c9
Fix
dsaxton Oct 2, 2020
eabef62
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 2, 2020
72be97f
Fix and test
dsaxton Oct 2, 2020
e1d1ba3
Fix
dsaxton Oct 2, 2020
d38839c
Fix
dsaxton Oct 2, 2020
0da1c46
Nit
dsaxton Oct 2, 2020
4ea7dcf
Nit
dsaxton Oct 2, 2020
376b06e
Change doc
dsaxton Oct 2, 2020
473d674
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 2, 2020
e564fee
Escape
dsaxton Oct 2, 2020
6cec317
Add failing test
dsaxton Oct 2, 2020
50ccbc0
Edit
dsaxton Oct 2, 2020
e725cb2
Maybe
dsaxton Oct 2, 2020
3a94daa
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 2, 2020
03e8cab
Comment
dsaxton Oct 2, 2020
46949fd
Remove
dsaxton Oct 2, 2020
11f20d0
Compile and comment
dsaxton Oct 3, 2020
44f0792
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 3, 2020
229b5fc
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 3, 2020
eec9d89
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 8, 2020
0e0dd92
Move into other function
dsaxton Oct 8, 2020
956ecf3
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 8, 2020
25dd8f1
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 10, 2020
eca0413
Note
dsaxton Oct 11, 2020
00b71b2
Merge remote-tracking branch 'upstream/master' into fix-is-numeric-he…
dsaxton Oct 11, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.2.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -424,6 +424,7 @@ I/O
- Bug in :func:`read_table` and :func:`read_csv` when ``delim_whitespace=True`` and ``sep=default`` (:issue:`36583`)
- Bug in :meth:`to_json` with ``lines=True`` and ``orient='records'`` the last line of the record is not appended with 'new line character' (:issue:`36888`)
- Bug in :meth:`read_parquet` with fixed offset timezones. String representation of timezones was not recognized (:issue:`35997`, :issue:`36004`)
- Bug in output rendering of complex numbers showing too many trailing zeros (:issue:`36799`)

Plotting
^^^^^^^^
Expand Down
10 changes: 5 additions & 5 deletions pandas/core/frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -2659,11 +2659,11 @@ def memory_usage(self, index=True, deep=False) -> Series:
>>> df = pd.DataFrame(data)
>>> df.head()
int64 float64 complex128 object bool
0 1 1.0 1.000000+0.000000j 1 True
1 1 1.0 1.000000+0.000000j 1 True
2 1 1.0 1.000000+0.000000j 1 True
3 1 1.0 1.000000+0.000000j 1 True
4 1 1.0 1.000000+0.000000j 1 True
0 1 1.0 1.0+0.0j 1 True
1 1 1.0 1.0+0.0j 1 True
2 1 1.0 1.0+0.0j 1 True
3 1 1.0 1.0+0.0j 1 True
4 1 1.0 1.0+0.0j 1 True

>>> df.memory_usage()
Index 128
Expand Down
31 changes: 22 additions & 9 deletions pandas/io/formats/format.py
Original file line number Diff line number Diff line change
Expand Up @@ -1474,9 +1474,9 @@ def format_values_with(float_format):

if self.fixed_width:
if is_complex:
result = _trim_zeros_complex(values, self.decimal, na_rep)
result = _trim_zeros_complex(values, self.decimal)
else:
result = _trim_zeros_float(values, self.decimal, na_rep)
result = _trim_zeros_float(values, self.decimal)
return np.asarray(result, dtype="object")

return values
Expand Down Expand Up @@ -1857,29 +1857,42 @@ def just(x):
return result


def _trim_zeros_complex(
str_complexes: np.ndarray, decimal: str = ".", na_rep: str = "NaN"
) -> List[str]:
def _trim_zeros_complex(str_complexes: np.ndarray, decimal: str = ".") -> List[str]:
"""
Separates the real and imaginary parts from the complex number, and
executes the _trim_zeros_float method on each of those.
"""
return [
"".join(_trim_zeros_float(re.split(r"([j+-])", x), decimal, na_rep))
trimmed = [
"".join(_trim_zeros_float(re.split(r"([j+-])", x), decimal))
for x in str_complexes
]

# pad strings to the length of the longest trimmed string for alignment
lengths = [len(s) for s in trimmed]
max_length = max(lengths)
padded = [
s[: -((k - 1) // 2 + 1)] # real part
+ (max_length - k) // 2 * "0"
+ s[-((k - 1) // 2 + 1) : -((k - 1) // 2)] # + / -
+ s[-((k - 1) // 2) : -1] # imaginary part
+ (max_length - k) // 2 * "0"
+ s[-1]
for s, k in zip(trimmed, lengths)
]
return padded


def _trim_zeros_float(
str_floats: Union[np.ndarray, List[str]], decimal: str = ".", na_rep: str = "NaN"
str_floats: Union[np.ndarray, List[str]], decimal: str = "."
) -> List[str]:
"""
Trims zeros, leaving just one before the decimal points if need be.
"""
trimmed = str_floats
number_regex = re.compile(fr"\s*[\+-]?[0-9]+(\{decimal}[0-9]*)?")

def _is_number(x):
return x != na_rep and not x.endswith("inf")
return re.match(number_regex, x) is not None

def _cond(values):
finite = [x for x in values if _is_number(x)]
Expand Down
7 changes: 7 additions & 0 deletions pandas/tests/io/formats/test_format.py
Original file line number Diff line number Diff line change
Expand Up @@ -3432,3 +3432,10 @@ def test_format_remove_leading_space_dataframe(input_array, expected):
# GH: 24980
df = pd.DataFrame(input_array).to_string(index=False)
assert df == expected


def test_to_string_complex_number_trims_zeros():
s = pd.Series([1.000000 + 1.000000j, 1.0 + 1.0j, 1.05 + 1.0j])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why should these have 2 decimal zeros and not 1 likely ordinary floats? Where did you get the expected output from?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a 1.05 in the last element, the expected output is similar to what happens for floats:

[ins] In [2]: s = pd.Series([1.0, 1.0000, 1.05])

[ins] In [3]: s
Out[3]: 
0    1.00
1    1.00
2    1.05
dtype: float64

result = s.to_string()
expected = "0 1.00+1.00j\n1 1.00+1.00j\n2 1.05+1.00j"
assert result == expected