PERF: MultiIndex.union

[lukemanley]

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v1.6.0.rst file if fixing a bug or adding a new feature.

cc @phofl - following your improvements to MultiIndex.difference, it looks like we can replace fast_unique_multiple.

     [57630237]       [7d8e3cf8]
     <main>           <multiindex-union>
-      80.0±0.9ms       61.7±0.6ms     0.77  multiindex_object.SetOperations.time_operation('non_monotonic', 'string', 'union')
-        81.7±1ms         61.1±1ms     0.75  multiindex_object.SetOperations.time_operation('monotonic', 'string', 'union')
-      55.6±0.4ms       38.3±0.3ms     0.69  multiindex_object.SetOperations.time_operation('non_monotonic', 'ea_int', 'union')
-        53.8±1ms       36.6±0.8ms     0.68  multiindex_object.SetOperations.time_operation('monotonic', 'ea_int', 'union')
-      34.1±0.4ms       14.2±0.6ms     0.42  multiindex_object.SetOperations.time_operation('non_monotonic', 'int', 'union')
-        33.9±2ms       13.8±0.6ms     0.41  multiindex_object.SetOperations.time_operation('monotonic', 'int', 'union')
-        75.8±4ms       14.9±0.4ms     0.20  multiindex_object.SetOperations.time_operation('non_monotonic', 'datetime', 'union')
-        75.4±3ms       14.3±0.6ms     0.19  multiindex_object.SetOperations.time_operation('monotonic', 'datetime', 'union')
- ```

[phofl]

How does this behave with duplicates and especially with Indexes that contain tuples?

Apart from that, I would prefer not doing that now. I have a different pr that I would prefer merging first.

[lukemanley]

Duplicates shouldn’t reach this given the if else block this is within (other is always unique). Not sure about tuples but I can take a look. Or I can just close it if you have something in the works.

[phofl]

Not really in the works yet, but I would prefer optimising the duplicates in other case before we optimise here. This is more needed (or you could look into that if you like, I suspect that we have to push that into the cython layer to get an acceptable performance improvement)

[lukemanley]

Sure, I'll take a look at the duplicates. I'll leave this open a bit longer.

[lukemanley]

@phofl, are you open to this one now that the duplicates case has been optimized a bit?

[phofl]

Can you rebase now?

[phofl]

thx @lukemanley