BUG: do not replace all nulls with "NaN"-string in Series index #45283

realead · 2022-01-09T06:58:27Z

closes BUG: In Series all null-values are printed as NaN #45263, BUG: value counts with mixed NaNs is oddly rendering #45222
tests added / passed
Ensure all linting tests pass, see here for how to run them
whatsnew entry

After this, the example

import pandas as pd
import numpy as np

s=pd.Series([1,2,3,4], [True, None, np.nan, pd.NaT])

print(s)

yields

True    1
None    2
nan     3
NaT     4
dtype: int64

which is good.

I've kept the behavioral changes minimal, however not sure it is the best way to go.

SeriesFormatter has a na_rep (

pandas/pandas/io/formats/format.py

Line 280 in b77797c

self.na_rep = na_rep

) but it is not used for index-formating (I use None to signal that I don't want any replacing). But why should values be handled differently?

Why does is make sense to replace nans with na_repr in the index in the first place? See

pandas/pandas/core/indexes/base.py

Lines 1392 to 1411 in b77797c

    
           def _format_with_header(self, header: list[str_t], na_rep: str_t) -> list[str_t]: 
        
               from pandas.io.formats.format import format_array 
        
               values = self._values 
        
               if is_object_dtype(values.dtype): 
        
                   values = cast(np.ndarray, values) 
        
                   values = lib.maybe_convert_objects(values, safe=True) 
        
                   result = [pprint_thing(x, escape_chars=("\t", "\r", "\n")) for x in values] 
        
                   # could have nans 
        
                   mask = isna(values) 
        
                   if mask.any(): 
        
                       result_arr = np.array(result) 
        
                       result_arr[mask] = na_rep 
        
                       result = result_arr.tolist() 
        
               else: 
        
                   result = trim_front(format_array(values, None, justify="left")) 
        
               return header + result

pep8speaks · 2022-01-09T06:58:31Z

Hello @realead! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-01-19 07:01:19 UTC

realead · 2022-01-09T07:13:55Z

This is the commit where special handling for the null-objects was added: in ##3034 with aae6213

pandas/tests/series/test_repr.py

realead · 2022-01-13T06:41:53Z

The question is what to do about DataFrame?

import pandas as pd
import numpy as np
s=pd.DataFrame([1, 2, 3, 4], [True, None, np.nan, pd.NaT])
print(s)

leads to

      0
True  1
NaN   2
NaN   3
NaN   4

It probably should behave the same way Series behaves?

jreback · 2022-01-13T12:08:01Z

yes

realead · 2022-01-13T21:21:05Z

After the change np.nan is printed as "nan". Probably we would like to keep "NaN"?

jreback · 2022-01-16T16:52:58Z

pandas/io/formats/format.py

@@ -366,10 +366,10 @@ def _get_formatted_index(self) -> tuple[list[str], bool]:

        if isinstance(index, MultiIndex):
            have_header = any(name for name in index.names)
-            fmt_index = index.format(names=True)
+            fmt_index = index.format(names=True, na_rep=None)


i think we need to correctly format floats with NaN in the indexes itself (if nothing is passed) to have NaN (rather than nan). see if that works.

i think we need to correctly format floats with NaN in the indexes itself (if nothing is passed) to have NaN (rather than nan). see if that works.

@jreback True, it seems like a better solution now.

There are some test failures now, but it looks like not due to this change (maybe docs - but I don't see what went wrong there).

excellent! yeah let me see what's up with the docs

ErSheetal · 2022-01-17T08:48:41Z

How can we access values where index is NaN??
s[NaN]
s[pd.NaT]
results in an error
Traceback (most recent call last):
File "<pyshell#16>", line 1, in
s[NaN]
NameError: name 'NaN' is not defined

pandas/_libs/missing.pyx

jreback · 2022-01-18T23:11:34Z

@realead can you merge master. we fixed the doc links there so let's see

realead · 2022-01-19T11:08:41Z

@jreback it seems to work now.

jreback · 2022-01-19T14:06:41Z

pandas/_libs/missing.pyx

@@ -248,6 +248,31 @@ cdef bint checknull_with_nat_and_na(object obj):
    return checknull_with_nat(obj) or obj is C_NA


+@cython.wraparound(False)


ok not super happy we need this but ok. cc @jbrockmendel

jreback

thanks @realead

…"-string in Series index

…n Series index (#45473) Co-authored-by: realead <[email protected]>

realead mentioned this pull request Jan 9, 2022

DOC: improving example #45262

Closed

1 task

realead requested a review from jreback January 9, 2022 07:19

mroeschke reviewed Jan 9, 2022

View reviewed changes

pandas/tests/series/test_repr.py Outdated Show resolved Hide resolved

jreback added this to the 1.4 milestone Jan 10, 2022

jreback added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Output-Formatting __repr__ of pandas objects, to_string labels Jan 10, 2022

realead changed the title ~~WIP: [BUG] do not replace all nulls with "NaN"-string in Series index~~ [BUG] do not replace all nulls with "NaN"-string in Series index Jan 10, 2022

realead force-pushed the fix_45263 branch from ff136fe to 79c49bc Compare January 12, 2022 13:12

realead changed the title ~~[BUG] do not replace all nulls with "NaN"-string in Series index~~ BUG: do not replace all nulls with "NaN"-string in Series index Jan 13, 2022

realead force-pushed the fix_45263 branch from 5568278 to 57c4148 Compare January 13, 2022 05:42

jreback reviewed Jan 16, 2022

View reviewed changes

realead force-pushed the fix_45263 branch 2 times, most recently from 02eec99 to 12c1c4a Compare January 18, 2022 14:43

mroeschke reviewed Jan 18, 2022

View reviewed changes

pandas/_libs/missing.pyx Outdated Show resolved Hide resolved

mroeschke reviewed Jan 18, 2022

View reviewed changes

pandas/_libs/missing.pyx Show resolved Hide resolved

mroeschke approved these changes Jan 18, 2022

View reviewed changes

realead added 7 commits January 19, 2022 08:00

only replace float-nans with NaN and not other null objects

b02d813

adding examples with series

7b73be6

adding examples with dataframes

82b0a37

adding what is new

11572d8

fixing test cases

cb6c0e7

adding def to pyi-file

ac460d1

fix intendation

0877371

realead force-pushed the fix_45263 branch from d071bb5 to 0877371 Compare January 19, 2022 07:01

jreback reviewed Jan 19, 2022

View reviewed changes

jreback approved these changes Jan 19, 2022

View reviewed changes

jreback merged commit b8cce91 into pandas-dev:main Jan 19, 2022

meeseeksmachine mentioned this pull request Jan 19, 2022

Backport PR #45283 on branch 1.4.x (BUG: do not replace all nulls with "NaN"-string in Series index) #45473

Merged

meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Jan 19, 2022

Backport PR pandas-dev#45283: BUG: do not replace all nulls with "NaN…

54c3e83

…"-string in Series index

jreback pushed a commit that referenced this pull request Jan 19, 2022

Backport PR #45283: BUG: do not replace all nulls with "NaN"-string i…

aaba0ef

…n Series index (#45473) Co-authored-by: realead <[email protected]>

realead mentioned this pull request Jan 20, 2022

BUG: value counts with mixed NaNs is oddly rendering #45222

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: do not replace all nulls with "NaN"-string in Series index #45283

BUG: do not replace all nulls with "NaN"-string in Series index #45283

realead commented Jan 9, 2022 •

edited

Loading

pep8speaks commented Jan 9, 2022 •

edited

Loading

realead commented Jan 9, 2022 •

edited

Loading

realead commented Jan 13, 2022

jreback commented Jan 13, 2022

realead commented Jan 13, 2022

jreback Jan 16, 2022

realead Jan 18, 2022

jreback Jan 18, 2022

ErSheetal commented Jan 17, 2022

jreback commented Jan 18, 2022

realead commented Jan 19, 2022

jreback Jan 19, 2022

jreback left a comment •

edited

Loading

	def _format_with_header(self, header: list[str_t], na_rep: str_t) -> list[str_t]:
	from pandas.io.formats.format import format_array

	values = self._values

	if is_object_dtype(values.dtype):
	values = cast(np.ndarray, values)
	values = lib.maybe_convert_objects(values, safe=True)

	result = [pprint_thing(x, escape_chars=("\t", "\r", "\n")) for x in values]

	# could have nans
	mask = isna(values)
	if mask.any():
	result_arr = np.array(result)
	result_arr[mask] = na_rep
	result = result_arr.tolist()
	else:
	result = trim_front(format_array(values, None, justify="left"))
	return header + result

		@@ -248,6 +248,31 @@ cdef bint checknull_with_nat_and_na(object obj):
		return checknull_with_nat(obj) or obj is C_NA


		@cython.wraparound(False)

BUG: do not replace all nulls with "NaN"-string in Series index #45283

BUG: do not replace all nulls with "NaN"-string in Series index #45283

Conversation

realead commented Jan 9, 2022 • edited Loading

pep8speaks commented Jan 9, 2022 • edited Loading

Comment last updated at 2022-01-19 07:01:19 UTC

realead commented Jan 9, 2022 • edited Loading

realead commented Jan 13, 2022

jreback commented Jan 13, 2022

realead commented Jan 13, 2022

jreback Jan 16, 2022

Choose a reason for hiding this comment

realead Jan 18, 2022

Choose a reason for hiding this comment

jreback Jan 18, 2022

Choose a reason for hiding this comment

ErSheetal commented Jan 17, 2022

jreback commented Jan 18, 2022

realead commented Jan 19, 2022

jreback Jan 19, 2022

Choose a reason for hiding this comment

jreback left a comment • edited Loading

Choose a reason for hiding this comment

realead commented Jan 9, 2022 •

edited

Loading

pep8speaks commented Jan 9, 2022 •

edited

Loading

realead commented Jan 9, 2022 •

edited

Loading

jreback left a comment •

edited

Loading