BUG: True cannot be cast to bool in read_excel #58994

rmhowe425 · 2024-06-13T02:17:20Z

closes BUG: True cannot be cast to bool #58159
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

rmhowe425 · 2024-06-13T03:40:19Z

@rhshadrach pinging on green

asishm · 2024-06-13T05:45:23Z

pandas/tests/io/excel/test_readers.py

@@ -164,6 +164,17 @@ def xfail_datetimes_with_pyxlsb(engine, request):


 class TestReaders:
+    def test_read_excel_type_check(self):
+        # GH 58159
+        df = DataFrame({"bool_column": [True]}, dtype=pd.BooleanDtype.name)


since boolean is a nullable EA, can you also add None as a value in this column? the assert can also probably change to tm.assert_frame_equal(df, df2)

rmhowe425 · 2024-06-15T23:42:31Z

@rhshadrach @asishm

PR ready for review. Not sure what's going on with the lone failing test. It's passing locally. Not much info from the test's output to go on.

asishm

Test failure is unrelated - #59018 (comment)

just one comment - may not even be relevant.

asishm · 2024-06-16T01:03:35Z

pandas/io/parsers/base_parser.py

+                        values_str,
                        dtype=cast_type,
                        true_values=self.true_values,
                        false_values=self.false_values,


none_values is not passed anything here. Can the equivalent na_values and keep_default_na be passed in here, or is it irrelevant?

I think I see what you're saying. You're suggesting to propagate na_values and keep_default_na from read_excel() to _cast_types()?

I think that's a good idea. I'll make that change. That should allow me to get rid of _NONE_VALUES in boolean.py, and allow me to dynamically handle any values that equate to None.

Those params are already available in _cast_types since it's an instance method (so self.na_values and self.keep_default_na are available here, similar to self.true_values and self.false_values). I was thinking about progagating them to _from_sequence_of_strings()

Yep we're on the same page. Making those changes now

… unit tests

rmhowe425 · 2024-06-17T02:22:49Z

@asishm @rhshadrach PR is ready for review

mroeschke · 2024-06-18T18:00:59Z

doc/source/whatsnew/v3.0.0.rst

@@ -547,6 +547,7 @@ I/O
 - Bug in :meth:`DataFrame.to_stata` when writing :class:`DataFrame` and ``byteorder=`big```. (:issue:`58969`)
 - Bug in :meth:`DataFrame.to_string` that raised ``StopIteration`` with nested DataFrames. (:issue:`16098`)
 - Bug in :meth:`read_csv` raising ``TypeError`` when ``index_col`` is specified and ``na_values`` is a dict containing the key ``None``. (:issue:`57547`)
+- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype=pd.BooleanDtype.name``. (:issue:`58159`)


Suggested change

- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype=pd.BooleanDtype.name``. (:issue:`58159`)

- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype="boolean"``. (:issue:`58159`)

mroeschke · 2024-06-18T18:01:09Z

pandas/core/arrays/boolean.py

@@ -297,7 +297,13 @@ class BooleanArray(BaseMaskedArray):
    """

    _TRUE_VALUES = {"True", "TRUE", "true", "1", "1.0"}
-    _FALSE_VALUES = {"False", "FALSE", "false", "0", "0.0"}
+    _FALSE_VALUES = {


Could you undo the formatting here?

mroeschke · 2024-06-18T18:04:32Z

pandas/tests/io/excel/test_readers.py

+    def test_read_excel_type_check(self, col):
+        # GH 58159
+        df = DataFrame({"bool_column": col}, dtype=pd.BooleanDtype.name)
+        df.to_excel("test-type.xlsx", index=False)


Can you use the tmp_excel fixture for this path?

mroeschke · 2024-06-18T18:05:16Z

pandas/tests/io/excel/test_readers.py

+
+    def test_pass_none_type(self):
+        # GH 58159
+        with pd.ExcelFile("test7.xlsx") as excel:


Can you use the datapath fixture to access test7.xlsx. Also could you give a test7.xlsx a better name reflective of the contents?

mroeschke · 2024-06-18T18:06:49Z

pandas/tests/io/excel/test_readers.py

+    @pytest.mark.parametrize("col", [[True, None, False], [True], [True, False]])
+    def test_read_excel_type_check(self, col):
+        # GH 58159
+        df = DataFrame({"bool_column": col}, dtype=pd.BooleanDtype.name)


Suggested change

df = DataFrame({"bool_column": col}, dtype=pd.BooleanDtype.name)

df = DataFrame({"bool_column": col}, dtype="boolean")

And could you change other usages here?

… into dev/bug/cast-to-boolean

rmhowe425 · 2024-06-23T02:55:25Z

@mroeschke PR ready for review

mroeschke · 2024-06-25T17:28:49Z

pandas/tests/io/excel/test_readers.py

+        f_path = os.path.join(
+            datapath("io", "data", "excel"), "test_boolean_types.xlsx"
+        )


Suggested change

f_path = os.path.join(

datapath("io", "data", "excel"), "test_boolean_types.xlsx"

)

f_path = datapath("io", "data", "excel", "test_boolean_types.xlsx")

mroeschke · 2024-06-25T17:29:06Z

pandas/tests/io/excel/test_readers.py

+
+    def test_pass_none_type(self, datapath):
+        # GH 58159
+        f_path = os.path.join(datapath("io", "data", "excel"), "test_none_type.xlsx")


Suggested change

f_path = os.path.join(datapath("io", "data", "excel"), "test_none_type.xlsx")

f_path = datapath("io", "data", "excel", "test_none_type.xlsx")

mroeschke · 2024-06-25T20:25:06Z

Thanks @rmhowe425

rmhowe425 added 2 commits June 12, 2024 22:12

Adding implementation, tests. Updating documentation

1fcbde6

Fixing dependency error by moving test inside of TestReaders

f77c41d

asishm reviewed Jun 13, 2024

View reviewed changes

rmhowe425 added 4 commits June 15, 2024 14:56

Updating implementation based on reviewer feedback

ffb65d4

Creating a more clean implementation

14e565e

Fixing broken unit tests

873bc9b

Fixing docstring error

e918066

rmhowe425 requested a review from asishm June 15, 2024 23:41

asishm reviewed Jun 16, 2024

View reviewed changes

rmhowe425 and others added 2 commits June 16, 2024 21:06

Updating implementation based on reviewer feedback. Adding additional…

7de150e

… unit tests

Merge branch 'pandas-dev:main' into dev/bug/cast-to-boolean

e9fe60c

rmhowe425 requested a review from asishm June 17, 2024 02:22

mroeschke reviewed Jun 18, 2024

View reviewed changes

mroeschke added the IO Excel read_excel, to_excel label Jun 18, 2024

mroeschke reviewed Jun 18, 2024

View reviewed changes

rmhowe425 added 5 commits June 21, 2024 23:07

Updating implementation based on reviewer feedback

2a2e8a3

Merge branch 'dev/bug/cast-to-boolean' of github.com:rmhowe425/pandas…

561b6ff

… into dev/bug/cast-to-boolean

Updating implementation based on reviewer feedback

614c880

Using datapath fixture

c62571e

Fixing failing unit test

d871d79

rmhowe425 requested a review from mroeschke June 23, 2024 02:55

rmhowe425 and others added 2 commits June 22, 2024 23:54

Removing unneeded file

306ea32

Merge branch 'main' into dev/bug/cast-to-boolean

d11f77b

Fixing failing documentation test

66d34f0

mroeschke reviewed Jun 25, 2024

View reviewed changes

Updating unit test based on reviewer feedback

42285ff

mroeschke approved these changes Jun 25, 2024

View reviewed changes

mroeschke added this to the 3.0 milestone Jun 25, 2024

mroeschke merged commit 1a95c79 into pandas-dev:main Jun 25, 2024
44 checks passed

rmhowe425 deleted the dev/bug/cast-to-boolean branch June 25, 2024 20:32

asishm mentioned this pull request Jun 25, 2024

BUG: Reading a Boolean column with "blanks" from an Excel file raises an absurd error #45903

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: True cannot be cast to bool in read_excel #58994

BUG: True cannot be cast to bool in read_excel #58994

rmhowe425 commented Jun 13, 2024 •

edited

Loading

rmhowe425 commented Jun 13, 2024

asishm Jun 13, 2024 •

edited

Loading

rmhowe425 commented Jun 15, 2024

asishm left a comment •

edited

Loading

asishm Jun 16, 2024

rmhowe425 Jun 16, 2024 •

edited

Loading

asishm Jun 16, 2024

rmhowe425 Jun 16, 2024

rmhowe425 commented Jun 17, 2024

mroeschke Jun 18, 2024

mroeschke Jun 18, 2024

mroeschke Jun 18, 2024

mroeschke Jun 18, 2024

mroeschke Jun 18, 2024

rmhowe425 commented Jun 23, 2024

mroeschke Jun 25, 2024

mroeschke Jun 25, 2024

mroeschke commented Jun 25, 2024

	- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype=pd.BooleanDtype.name``. (:issue:`58159`)
	- Bug in :meth:`read_excel` raising ``ValueError`` when passing array of boolean values when ``dtype="boolean"``. (:issue:`58159`)

	df = DataFrame({"bool_column": col}, dtype=pd.BooleanDtype.name)
	df = DataFrame({"bool_column": col}, dtype="boolean")

	f_path = os.path.join(datapath("io", "data", "excel"), "test_none_type.xlsx")
	f_path = datapath("io", "data", "excel", "test_none_type.xlsx")

BUG: True cannot be cast to bool in read_excel #58994

BUG: True cannot be cast to bool in read_excel #58994

Conversation

rmhowe425 commented Jun 13, 2024 • edited Loading

rmhowe425 commented Jun 13, 2024

asishm Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

rmhowe425 commented Jun 15, 2024

asishm left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmhowe425 Jun 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmhowe425 commented Jun 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmhowe425 commented Jun 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mroeschke commented Jun 25, 2024

rmhowe425 commented Jun 13, 2024 •

edited

Loading

asishm Jun 13, 2024 •

edited

Loading

asishm left a comment •

edited

Loading

rmhowe425 Jun 16, 2024 •

edited

Loading