ENH: `Styler.apply(map)_index` made compatible with `Styler.to_excel` #41995

attack68 · 2021-06-14T09:01:41Z

This is a follow-on to #41893.

It makes the new Styler.apply_index and Styler.applymap_index methods compatible with Styler.to_excel. If the methods are not used the output reverts to the old format.

MultiIndex Case

midx = pd.MultiIndex.from_product([['ix', 'jy'], [0, 1], ['x3', 'z4']])
df = pd.DataFrame([np.arange(8)], columns=midx)
def highlight_x(s):
    return ["background-color: yellow;" if 'x' in v else "" for v in s]
df.style.apply_index(highlight_x, axis="columns", level=[0, 2]).to_excel("mi_test.xlsx")

NOTE on TESTS

The tests for styler_to_excel were broken and were xfailed out of production in 2019. Since they dont work I have replaced them with some other, cleaner, tests and add tests relevant to this PR also.

Closes #25351

…x_specific

…x_specific # Conflicts: # doc/source/reference/style.rst # pandas/io/formats/style_render.py # pandas/tests/io/formats/style/test_html.py

…ecific_excel

…x_specific # Conflicts: # pandas/tests/io/formats/style/test_style.py

…ecific_excel

…x_specific_excel

…x_specific_excel # Conflicts: # doc/source/whatsnew/v1.4.0.rst

jreback · 2021-09-09T12:36:48Z

pandas/io/formats/excel.py

@@ -472,6 +472,7 @@ def __init__(
        self.na_rep = na_rep
        if not isinstance(df, DataFrame):
            self.styler = df
+            self.styler._compute()


why is this needed? init doesn't do anything really except for set things

this clause is only called when you do styler.to_excel. It needs _compute to calculate the styles added. Previously, this was done in _generate_body below, but, since, it is now needed for both body and headers I brought it to the obvious point: initialisation, to avoid having to unnecessarily do it twice.

could maybe move this to styler.to_excel and maybe do the pre-calc there before passing to ExcelFormatter. Might be more sensible like that.

could maybe move this to styler.to_excel and maybe do the pre-calc there before passing to ExcelFormatter. Might be more sensible like that.

I tried it and it didn't work. The new tests failed when using a custom ExcelFormatter, the styles computation needs to be done within ExcelFormatter.

…x_specific_excel

attack68 · 2021-09-25T19:47:53Z

I discovered #25351 where basically the tests for Styler.to_excel had all been removed, and were pretty ugly anyway.

So I have replaced these with a minimalist set of non-comprehensive tests, and also added the tests relevant to this PR.

jreback · 2021-09-28T21:02:55Z

pandas/io/formats/excel.py

@@ -582,7 +589,10 @@ def _format_header_mi(self) -> Iterable[ExcelCell]:
            # Format in legacy format with dots to indicate levels.
            for i, values in enumerate(zip(*level_strs)):
                v = ".".join(map(pprint_thing, values))
-                yield ExcelCell(lnum, coloffset + i + 1, v, self.header_style)
+                if styles:


worth making a function to do this conversion as this is repeated below

jreback · 2021-09-28T21:03:44Z

pandas/tests/io/excel/test_style.py

@@ -7,163 +7,158 @@
 from pandas.io.excel import ExcelWriter
 from pandas.io.formats.excel import ExcelFormatter

+pytest.importorskip("jinja2")


this is essentially a new requirement for this right? its ok ,but let's fully document this change.

so, actually this file only had one test and that test already had importskip("jinja2"), so I haven't added anything, just moved it top level to share across all tests.

I noticed, however, that technically it is would be possible to create a Styler, apply methods and then do .to_excel all without jinja2. The only problem is that Styler does a check for jinja2 at initialisation, so this is not currently functionally possible.

its fine to require jinja2 for this, just mention this in the docs / doc-string somewhere. (pretty sure we should be testing this but pls ensure somewhere (and skipping elsewhere)

there is an import error warning if jinja2 not present. But could be more explicit. Will do as a follow up.

…x_specific_excel

jreback · 2021-10-03T22:18:19Z

pandas/io/formats/excel.py

@@ -472,12 +493,14 @@ def __init__(
        self.na_rep = na_rep
        if not isinstance(df, DataFrame):
            self.styler = df
+            self.styler._compute()  # calculate applied styles


why does this need to be called? e.g. shouldn't this be in render?

#41995 (comment)

ok i see your expl. i guess this is fine.

jreback · 2021-10-03T22:19:14Z

pandas/tests/io/excel/test_style.py

@@ -7,163 +7,158 @@
 from pandas.io.excel import ExcelWriter
 from pandas.io.formats.excel import ExcelFormatter

+pytest.importorskip("jinja2")


its fine to require jinja2 for this, just mention this in the docs / doc-string somewhere. (pretty sure we should be testing this but pls ensure somewhere (and skipping elsewhere)

attack68 · 2021-10-05T16:40:57Z

@jreback ping green

…x_specific_excel

…x_specific_excel # Conflicts: # doc/source/whatsnew/v1.4.0.rst

…x_specific_excel

jreback · 2021-10-19T00:43:37Z

pandas/io/formats/excel.py

@@ -472,12 +493,14 @@ def __init__(
        self.na_rep = na_rep
        if not isinstance(df, DataFrame):
            self.styler = df
+            self.styler._compute()  # calculate applied styles


ok i see your expl. i guess this is fine.

jreback · 2021-10-19T00:44:27Z

thanks @attack68

attack68 added 20 commits June 9, 2021 14:36

add apply across index

ece8855

add applymap across index

a3a88e5

improve docs

066e4f3

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

50dbcc7

…x_specific

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

ef5839f

…x_specific

add column header styling and amend tests

5e4c1c0

doc sharing

20ac7e0

doc fix

26c5340

doc fix

2437b72

collapse the cellstyle maps

312a6e6

add basic test

553426f

add basic test

f01dfee

parametrise tests

f940165

test for raises ValueError

6f5b46c

test html working

75cf6ca

test html working

d654139

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

728091a

…x_specific

whats new 1.4.0

17787ef

make apply_header compatible with to_excel

e2240f9

whats new

6a198d2

attack68 changed the title ~~Styler apply index specific excel~~ [WIP] ENH: Styler.apply(map)_header made compatible with Styler.to_excel (for non-MultiIndex) Jun 14, 2021

attack68 added 2 commits June 20, 2021 20:34

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

248de0d

…x_specific # Conflicts: # doc/source/reference/style.rst # pandas/io/formats/style_render.py # pandas/tests/io/formats/style/test_html.py

Merge branch 'styler_apply_index_specific' into styler_apply_index_sp…

8bc3e53

…ecific_excel

attack68 mentioned this pull request Jun 28, 2021

BUG: Styler.to_excel does not export styles and formats correctly #42276

Open

5 tasks

attack68 added 2 commits June 29, 2021 16:21

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

321458e

…x_specific # Conflicts: # pandas/tests/io/formats/style/test_style.py

Merge branch 'styler_apply_index_specific' into styler_apply_index_sp…

c104c3d

…ecific_excel

attack68 added Enhancement IO Excel read_excel, to_excel Styler conditional formatting using DataFrame.style labels Jun 29, 2021

add tests for new cases

13b5fe5

attack68 added 3 commits August 31, 2021 06:45

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

0832cbe

…x_specific_excel

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

c495aae

…x_specific_excel

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

96c6261

…x_specific_excel # Conflicts: # doc/source/whatsnew/v1.4.0.rst

jreback requested changes Sep 21, 2021

View reviewed changes

attack68 added 2 commits September 22, 2021 18:58

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

889d794

…x_specific_excel

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

79eb10c

…x_specific_excel

attack68 mentioned this pull request Sep 25, 2021

ERR: excel styler tests failing recently #25351

Closed

excel styler tests

d3fd856

attack68 added 2 commits September 26, 2021 09:12

add tests

702a2e0

add tests

77704cb

jreback requested changes Sep 28, 2021

View reviewed changes

attack68 added 6 commits October 1, 2021 17:29

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

cd9fe8b

…x_specific_excel

function

fe42e80

function

6391576

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

ab5e6a0

…x_specific_excel

new CssExcelCell class

ba4340c

direct args

20d0063

attack68 requested a review from jreback October 3, 2021 14:34

jreback requested changes Oct 3, 2021

View reviewed changes

comment on styler import jinja2 for to_excel

0913da9

attack68 added 3 commits October 9, 2021 12:22

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

09600f8

…x_specific_excel

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

9f8b8fa

…x_specific_excel # Conflicts: # doc/source/whatsnew/v1.4.0.rst

Merge remote-tracking branch 'upstream/master' into styler_apply_inde…

04eebeb

…x_specific_excel

jreback approved these changes Oct 19, 2021

View reviewed changes

jreback added this to the 1.4 milestone Oct 19, 2021

jreback merged commit f2abbd9 into pandas-dev:master Oct 19, 2021

attack68 deleted the styler_apply_index_specific_excel branch October 19, 2021 05:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: `Styler.apply(map)_index` made compatible with `Styler.to_excel` #41995

ENH: `Styler.apply(map)_index` made compatible with `Styler.to_excel` #41995

attack68 commented Jun 14, 2021 •

edited

Loading

jreback Sep 9, 2021

attack68 Sep 21, 2021

attack68 Sep 21, 2021

attack68 Sep 25, 2021

attack68 commented Sep 25, 2021 •

edited

Loading

jreback Sep 28, 2021

jreback Sep 28, 2021

attack68 Sep 29, 2021

jreback Oct 3, 2021

attack68 Oct 4, 2021

jreback Oct 3, 2021

attack68 Oct 4, 2021

jreback Oct 19, 2021

jreback Oct 3, 2021

attack68 commented Oct 5, 2021

jreback Oct 19, 2021

jreback commented Oct 19, 2021

ENH: Styler.apply(map)_index made compatible with Styler.to_excel #41995

ENH: Styler.apply(map)_index made compatible with Styler.to_excel #41995

Conversation

attack68 commented Jun 14, 2021 • edited Loading

NOTE on TESTS

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

attack68 commented Sep 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

attack68 commented Oct 5, 2021

Choose a reason for hiding this comment

jreback commented Oct 19, 2021

ENH: `Styler.apply(map)_index` made compatible with `Styler.to_excel` #41995

ENH: `Styler.apply(map)_index` made compatible with `Styler.to_excel` #41995

attack68 commented Jun 14, 2021 •

edited

Loading

attack68 commented Sep 25, 2021 •

edited

Loading