PERF: concatenation of MultiIndexed objects (MultiIndex.append)

[lukemanley]

• closes PERF: potential room for optimizing concatenation of MultiIndex (MultiIndex.append) #53574
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v2.1.0.rst file if fixing a bug or adding a new feature.

cc: @jorisvandenbossche - nice find/suggestion.

Perf improvement for MultiIndex.append (and pd.concat for objects with MultiIndexes):

       before           after         ratio
       <main>          <mi-append>
-        67.3±2ms       5.09±0.1ms     0.08  multiindex_object.Append.time_append('datetime64[ns]')
-        68.5±1ms       5.10±0.1ms     0.07  multiindex_object.Append.time_append('int64')
-         147±5ms       5.40±0.3ms     0.04  multiindex_object.Append.time_append('string')

[jorisvandenbossche]

Thanks for picking this up! And it seems that all tests pass, so that's nice ;)

Do you think it would be worth testing this with more MIs? (not necessarily in the asvs, can also be just here in the PR) Or are we confident this should be faster (or at least not a big slowdown) for most cases?

[jorisvandenbossche]

Suggested change

	mi.codes[i],
	mi.levels[i],
	level_values,
	copy=False,
	mi.codes[i], mi.levels[i], level_values, copy=False

(styling nitpick, this can fit on one line I think)

[lukemanley]

updated, thanks

[jorisvandenbossche]

Do you think it would be worth testing this with more MIs? (not necessarily in the asvs, can also be just here in the PR) Or are we confident this should be faster (or at least not a big slowdown) for most cases?

Did a quick test with a case of complete unique values in the MI levels (so not even repetition within a level):

N = 100_000
idx1 = pd.MultiIndex.from_arrays([range(N), np.random.choice(list(string.ascii_letters), size=(N, 5)).astype(object).sum(axis=1)])
idx2 = pd.MultiIndex.from_arrays([range(N, N*2), np.random.choice(list(string.ascii_letters), size=(N, 5)).astype(object).sum(axis=1)])
%timeit idx1.append(idx2)

and even here the new algo is faster (just a smaller difference, "only" around 2x faster).
My gut feeling says that such a case of all unique level values is the "worse" case for the recoding-based implementation, so that seems to indicate we can assume this to be faster generally (in the end, also with the current implementation of combining the (unique) level values, those have to be encoded afterwards anyway, so recoding shouldn't be worse here).

[mroeschke]

Very nice thanks @lukemanley

[phofl]

@lukemanley could you add a test like in the linked dask issue?