BUG/PERF: MultiIndex.value_counts returning flat index

[lukemanley]

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v2.0.0.rst file if fixing a bug or adding a new feature.

I believe it is a bug that MultiIndex.value_counts returns a Series indexed by a flat index of tuples rather than a MultiIndex. By returning a flat index, we lose nullable dtypes, index names, etc.

This PR changes MultiIndex.value_counts to return a Series indexed by a MultiIndex.

This also has some perf improvements:

import pandas as pd
import numpy as np

mi = pd.MultiIndex.from_product([np.arange(1000), np.arange(1000)])

%timeit mi.value_counts()

974 ms ± 28.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <- main
182 ms ± 10.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) <- PR

[phofl]

Two things:

Can we send this down a different path directly from MultiIndex.value_counts and what happens when bins is not None?
Edit: Missed the normalise part at the end, so forget my comment about sending it down a different path

[topper-123]

LGTM, pending green.

[lukemanley]

Two things:

Can we send this down a different path directly from MultiIndex.value_counts and what happens when bins is not None? Edit: Missed the normalise part at the end, so forget my comment about sending it down a different path

I also thought this path made sense since algos.union_with_duplicates calls algos.value_counts and will benefit from the performance improvements. The value_counts calls are currently bottlenecks when calling union_with_duplicates with MultiIndexes.

bins is documented to only work for numeric data. Passing anything other than None for a MultiIndex raises - this PR does not change that behavior. The error message could be improved as it simply raises when trying to construct a Series from a MultiIndex.

[phofl]

Thx, could you open an issue about improving the error message?

[mroeschke]

Suggested change

	- Bug in :meth:`MultiIndex.value_counts` returning a :class:`Series` indexed by flat index of tuples (:issue:`49558`)
	- Bug in :meth:`MultiIndex.value_counts` returning a :class:`Series` indexed by flat index of tuples instead of a :class:`MultiIndex` (:issue:`49558`)

Nit

[lukemanley]

updated, thanks

[lukemanley]

Thx, could you open an issue about improving the error message?

will do

[mroeschke]

LGTM merge when ready @phofl

[phofl]

thx @lukemanley