Skip to content

BUG:error in reading a boolean column with blanks from excel file #45903 #46374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

BUG:error in reading a boolean column with blanks from excel file #45903 #46374

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
532a846
BUG:error in reading a boolean column with blanks
dimitra-karadima Mar 15, 2022
4d2a6f5
TST: add xlsb file because of failing test
dimitra-karadima Mar 15, 2022
cf32733
Update doc/source/whatsnew/v1.5.0.rst
dimitra-karadima Apr 11, 2022
9d55372
Merge branch 'main' into fix-45903
dimitra-karadima Apr 11, 2022
5af0af4
TST: update unit tests for bool with blank
dimitra-karadima Apr 11, 2022
00d2195
Merge branch 'fix-45903' of https://github.com/dimitra-karadima/panda…
dimitra-karadima Apr 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.5.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -569,6 +569,7 @@ I/O
- Bug in Parquet roundtrip for Interval dtype with ``datetime64[ns]`` subtype (:issue:`45881`)
- Bug in :func:`read_excel` when reading a ``.ods`` file with newlines between xml elements (:issue:`45598`)
- Bug in :func:`read_parquet` when ``engine="fastparquet"`` where the file was not closed on error (:issue:`46555`)
- Bug in :func:`read_excel` when reading a ``.xlsx`` file with a boolean column that contains blank lines (:issue:`45903`)

Period
^^^^^^
Expand Down
5 changes: 3 additions & 2 deletions pandas/core/arrays/boolean.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,7 @@ class BooleanArray(BaseMaskedArray):
# Fill values used for any/all
_truthy_value = True
_falsey_value = False
_NONE_VALUES = {"nan", "NaN", "None", "NA", "null", "NULL"}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fail to understand how this is related to having blanks. In the current version of pandas (without the changes in this PR), what's the value of the blank value in the spreadsheet when constructing the extension array? Feels like it should be None or an empty string, not one of those.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@datapythonista Hello! Thanks for the review! The point here is that I added this list because the isna() fails. The None values are converted into strings so without this list, the "None" values will raise a ValueError. Of course if you have any suggestion on how to implement this better please tell me.

_TRUE_VALUES = {"True", "TRUE", "true", "1", "1.0"}
_FALSE_VALUES = {"False", "FALSE", "false", "0", "0.0"}

Expand Down Expand Up @@ -321,8 +322,8 @@ def _from_sequence_of_strings(
false_values_union = cls._FALSE_VALUES.union(false_values or [])

def map_string(s):
if isna(s):
return s
if isna(s) or s in cls._NONE_VALUES:
return None
elif s in true_values_union:
return True
elif s in false_values_union:
Expand Down
2 changes: 1 addition & 1 deletion pandas/io/parsers/base_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -785,7 +785,7 @@ def _cast_types(self, values, cast_type, column):
try:
if is_bool_dtype(cast_type):
return array_type._from_sequence_of_strings(
values,
values.astype(str),
dtype=cast_type,
true_values=self.true_values,
false_values=self.false_values,
Expand Down
Binary file modified pandas/tests/io/data/excel/test_types.ods
View file
Open in desktop
Binary file not shown.
Binary file modified pandas/tests/io/data/excel/test_types.xls
View file
Open in desktop
Binary file not shown.
Binary file modified pandas/tests/io/data/excel/test_types.xlsb
View file
Open in desktop
Binary file not shown.
Binary file modified pandas/tests/io/data/excel/test_types.xlsm
View file
Open in desktop
Binary file not shown.
Binary file modified pandas/tests/io/data/excel/test_types.xlsx
View file
Open in desktop
Binary file not shown.
2 changes: 2 additions & 0 deletions pandas/tests/io/excel/test_readers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,8 +420,10 @@ def test_reader_special_dtypes(self, request, read_ext):
datetime(2013, 12, 14),
datetime(2015, 3, 14),
],
"BoolColWithBlank": [True, False, None, True, False],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@datapythonista I upated the test_types files as you suggested but for some reason no False or True values are shown. Only 1 or 0 and I cannot find out why.

},
)

basename = "test_types"

# should read in correctly and infer types
Expand Down