BUG: Wrong dtype when resetting a multiindex with missing values. (#1…

[ldmnt]

…9602 and #24206)

• Fixed the bad path in reset_index where the dtype was ignored if there are only
  missing values.

• Simplified the structure of _maybe_casted_values in reset_index by
  using as much as possible the take method of ExtensionArray's

• closes reset_index() on MultiIndexed empty dataframe does not preserve dtypes  #19602

• closes BUG: reset_index of MultiIndex with CategoricalIndex levels with missing values fails #24206

• tests added / passed

• passes black pandas

• passes git diff upstream/master -u -- "*.py" | flake8 --diff

• whatsnew entry

[ldmnt]

Two possible improvements that i noticed while working on this :

1. The multiindex path of _maybe_casted_values is essentially a take(…, allow_fill=True) function that works on both ExtensionArrays and ndarrays. Since it's the type of the _values attribute, it might be a good idea to extract it outside of reset_index if it's used in other places. (I thought that it likely is but I didn't really know.)

2. When refactoring _maybe_casted_values in reset_index, I tried removing the cast at the start of the function (frame.py line 4607) :

            if not isinstance(index, (PeriodIndex, DatetimeIndex)):
                if values.dtype == np.object_:
                    values = lib.maybe_convert_objects(values)`

It was introduced a very long time ago (#440) to try and infer the type of indexes with dtype object. Removing it breaks one test that relies on the fact that an index with integer underlying data and object dtype will be correctly casted to an Int64Index. I didn't have enough knowledge of the code base to know if inferring the type of object indexes is still the intended behavior, but if it's not these lines could be removed.

[ldmnt]

@jreback for info: I have made all the modifications you requested up to now and on my side its ready. If you need further changes I can try to handle them before next friday, but after that I will be away for 3 weeks and unable to update the pull request.

[TomAugspurger]

Can you ensure that we have a basic test for #19602. Something like

In [11]: idx = pd.MultiIndex.from_product([[0, 1], [1, 2]])

In [12]: pd.DataFrame(index=idx)[:0].reset_index().dtypes
Out[12]:
level_0    float64
level_1    float64
dtype: object

asserting that those are ints, not floats?

[TomAugspurger]

I think mark this as private with a leading underscore.

[ldmnt]

Are you sure ? It's not really private since it's used in frame.py, and @jreback already asked me to remove the underscore when moving it.

[TomAugspurger]

Maybe wait to hear from @jreback but .core.* is supposed to be private, but people still use methods from there. We plan to deprecate it, but until then it's probably best to prefix methods with underscores.

[TomAugspurger]

Is it possible to get here with an extension dtype? Like pd.date_range('2000', periods=4, tz="CET")? If so, I suspect that will fail.

[ldmnt]

I didn't have time to really get into it, but after a quick look I would say that it's possible. I'll have to do something about it when i come back.

[jreback]

@ldmnt can you merge master and we'll take a look.

[jreback]

can you merge master

[ldmnt]

Sorry I haven't found time to do that recently. I saw your message, I will try to do the merge and think about Tom's question this week.

[WillAyd]

Thanks for the PR but I think this has gone stale. Closing to clean up the queue for now, but certainly ping if you'd like to pick back up