Conditional HTML styling hides MultiIndex structure

[mattilyra]

I noticed that the styling doesn't quite work with MultiIndex data frames. For example if we have a dataframe with two cross validated scores and take their meanand sem using groupby to get a MultiIndex dataframe

import itertools
import pandas as pd
import numpy as np

jobs = itertools.product(['a', 'b', 'c', 'd'], np.arange(1e-4, 1e-3, .0003), range(10))
rows = []
for v, v2, itr in jobs:
    rows.append({'param_1': v, 'param_2': v2, 'iter': itr,
                 'score_1': np.random.randint(0, 100, size=(1,))[0],
                 'score_2': np.random.rand(1, )[0]})
df_multi = pd.DataFrame(rows)
agg = df_multi.groupby(by=['param_1', 'param_2'])[['score_1', 'score_2']].agg(['mean', 'sem'])
agg.style.highlight_max()

where as without the styling the index names are preserved and the score_1 and score_2 column headers span 2 columns which improves legibility.

[jreback]

IIRC this is 'not implemented', cc @TomAugspurger as wanted to get basic stuff working.

patches are welcome!

[jreback]

see last check box in #11610

[TomAugspurger]

Yep. This one isn't too bad to do, I just ran out of time. It's a matter of modifying the template to be aware of the index levels.

[mattilyra]

@jreback ah I see, didn't read the post properly. I guess in that case this issue is redundant and can be closed.

[jreback]

@mattilyra its fine. you can post a fix and directly reference this.

[jreback]

this still open @TomAugspurger ?

[TomAugspurger]

Yeah, still open.

[jreback]

@TomAugspurger did you have a fix for this already? (even if not prod ready), just want to give a try.

[TomAugspurger]

@jreback I can get something together tonight, if that's not too late.

The basic idea is to use the logic in pandas.core.format._get_level_lengths on each of the MI elements. We'll put a couple more pieces of state in the internal dict is_visible, row_span and col_span.

[jreback]

that would be totally fantastic!

[TomAugspurger]

@jreback OK, the minimal version (1 whole test) is here: https://github.com/TomAugspurger/pandas/tree/style-sparse-mi-2

I'll see what refactoring I can do and submit a pull request sometime soonish.

[jreback]

hmm, doesn't seem to change the display. (using pandas 0.18.0 as a comparison). The column labels are not showing sparse (as what we normally do in an html display)

[TomAugspurger]

Ahh, only did index labels. Give me 20 minutes :)

[TomAugspurger]

@jreback pushed 4655534 which fix it.

https://gist.github.com/643402b241e6d945118942cb8ccff3ac

Index names are messed up with a MultiIndex, but if you just set those to "" then the None shouldn't appear.

[jreback]

that looks great! thanks!

I realize that the CSS injection can be done thru set_table_style I think as well.

[jreback]

@TomAugspurger so this fixes the single level index case (I don't have a good repro :<
but); also only barely tested on a single level columns (this works with the multi-level nicley as that was the point).

def _get_level_lengths(index):
    '''
    Given an index, find the level lenght for each element.

    Result is a dictionary of (level, inital_position): span
    '''
    sentinel = com.sentinel_factory()
    levels = index.format(sparsify=sentinel, adjoin=False, names=False)

    if index.nlevels == 1:
        return { (0, i):1 for i, value in enumerate(levels) }

    lengths = {}
    for i, lvl in enumerate(levels):
        for j, row in enumerate(lvl):
            if row != sentinel:
                last_label = j
                lengths[(i, last_label)] = 1
            else:
                lengths[(i, last_label)] += 1

    return lengths