BUG: Styling of a DataFrame with boolean column labels leaves out False and fails when adding non-numerical column

[bherwerth]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

# %%
import pandas as pd
from IPython import display

df = pd.DataFrame(
    [
        [1, 2],
        [3, 4]
    ],
    columns=[False, True],
    index=[1, 2],
)
# column `False` not styled (requires running in Jupyter)
display.HTML(df.style.background_gradient().to_html())
# %%
df["C"] = ["a", "b"]
# fails with IndexError
display.HTML(df.style.background_gradient().to_html())

Issue Description

Before adding C, the styled dataframe renders like this:

After adding C, I get IndexError: Boolean index has wrong length: 2 instead of 3.

Traceback

Traceback (most recent call last):
  File "minimal.py", line 18, in <module>
    display.HTML(df.style.background_gradient().to_html())
  File "/home/pandas/pandas/io/formats/style.py", line 1372, in to_html
    html = obj._render_html(
  File "/home/pandas/pandas/io/formats/style_render.py", line 202, in _render_html
    d = self._render(sparse_index, sparse_columns, max_rows, max_cols, "&nbsp;")
  File "/home/pandas/pandas/io/formats/style_render.py", line 166, in _render
    self._compute()
  File "/home/pandas/pandas/io/formats/style_render.py", line 254, in _compute
    r = func(self)(*args, **kwargs)
  File "/home/pandas/pandas/io/formats/style.py", line 1711, in _apply
    data = self.data.loc[subset]
  File "/home/pandas/pandas/core/indexing.py", line 1065, in __getitem__
    return self._getitem_tuple(key)
  File "/home/pandas/pandas/core/indexing.py", line 1254, in _getitem_tuple
    return self._getitem_tuple_same_dim(tup)
  File "/home/pandas/pandas/core/indexing.py", line 922, in _getitem_tuple_same_dim
    retval = getattr(retval, self.name)._getitem_axis(key, axis=i)
  File "/home/pandas/pandas/core/indexing.py", line 1287, in _getitem_axis
    return self._getbool_axis(key, axis=axis)
  File "/home/pandas/pandas/core/indexing.py", line 1089, in _getbool_axis
    key = check_bool_indexer(labels, key)
  File "/home/pandas/pandas/core/indexing.py", line 2561, in check_bool_indexer
    return check_array_indexer(index, result)
  File "/home/pandas/pandas/core/indexers/utils.py", line 552, in check_array_indexer
    raise IndexError(
IndexError: Boolean index has wrong length: 2 instead of 3

The same problem occurs when using style.bar instead of style.background_gradient.

Expected Behavior

First table:

After adding column "C":

I think the problem is that subset is by default taken to be the column names of the numerical columns of df

pandas/pandas/io/formats/style.py

Lines 2826 to 2827 in e8093ba

	if subset is None and gmap is None:
	subset = self.data.select_dtypes(include=np.number).columns

Later, df is indexed with loc, which treats subset as a boolean mask:

pandas/pandas/io/formats/style.py

Lines 1422 to 1423 in e8093ba

	subset = non_reducing_slice(subset)
	data = self.data.loc[subset]

A solution could be to make subset by default a boolean mask:

            subset = (
                self.data.columns.isin(
                    self.data.select_dtypes(include=np.number)
                )
            )

I tried this replacement and used it to produce the screenshots for the expected behavior above.

I suppose having boolean column labels is exotic and probably not a good idea generally. When indexing, for example df[[False]], pandas interprets this as a boolean mask. Yet, I encountered this problem when trying to visualize pd.crosstab(a, b), where b has boolean values.

If there is interest in fixing this, I'd be happy to work on a PR. Perhaps the very least, a warning should be shown when styling with boolean column labels.

Installed Versions

/opt/conda/lib/python3.8/site-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : a62897a
python : 3.8.13.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.104-linuxkit
Version : #1 SMP Thu Mar 17 17:08:06 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : C.UTF-8
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.5.0.dev0+1189.ga62897ae5f
numpy : 1.22.4
pytz : 2022.1
dateutil : 2.8.2
setuptools : 63.2.0
pip : 22.2
Cython : 0.29.30
pytest : 7.1.2
hypothesis : 6.47.1
sphinx : 4.5.0
blosc : None
feather : None
xlsxwriter : 3.0.3
lxml.etree : 4.9.1
html5lib : 1.1
pymysql : 1.0.2
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: 0.10.0
bs4 : 4.11.1
bottleneck : 1.3.5
brotli :
fastparquet : 0.8.1
fsspec : 2022.5.0
gcsfs : 2022.5.0
matplotlib : 3.5.2
numba : 0.55.2
numexpr : 2.8.3
odfpy : None
openpyxl : 3.0.9
pandas_gbq : 0.17.6
pyarrow : 8.0.0
pyreadstat : 1.1.9
pyxlsb : 1.0.9
s3fs : 0.6.0
scipy : 1.8.1
snappy :
sqlalchemy : 1.4.39
tables : 3.7.0
tabulate : 0.8.10
xarray : 2022.6.0
xlrd : 2.0.1
xlwt : 1.3.0
zstandard : 0.18.0

[attack68]

Yes this is a bit of a problem. The Styler subset arg is designed to work similar to .loc but doesn't always replicate it as you have discovered.

The solution proposed seems simple enough to possibly be included if it works and has no other impact.

I wonder if we document if something like: subset=(slice(None), slice(None)) can be used to overwrite the automatic selection in your case.

And you are right, mixing boolean names and boolean indexers is almost impossible to do smoothly in a package.

[bherwerth]

In the given case, I think there is no inconsistency between loc and subset given that subset is by default the column names of the numerical columns. If df has columns [False, True], then df.loc[:, [[False, True]] will only return the True column because boolean indexing is applied.

Still, I feel the behavoir of the styler is unexpected when it effectively does something like df.loc[:, df.columns] and thus removes the column False.

The suggested fix seems to work. Locally, I ran the tests in pandas/tests/io/formats/style/ and they still succeed after I implemented the change. I also added new tests, which fail without the change and succeed afterwards.

For a general df, I can think of two cases, where the way of indexing (df.loc[:, cols] vs df.loc[:, df.columns.isin(cols)]) would make a difference:

• If cols contains column names not present in df.
• If cols has another order than df.columns.
  Both should not be the case here because in our case cols is the output of df.select_dtypes(include=np.number).

Please le me know if there is any other check that you can think of that this change has no unintended consequences. I'll create a draft PR with my proposed solution later.