numpydev ragged array dtype warning

[TomAugspurger]

Closes #31201

There are still two failures in the Categorical constructor I'm looking into. Not sure what's best yet.

In [2]: pd.Categorical(['a', ('a', 'b')])
/Users/taugspurger/sandbox/pandas/pandas/core/dtypes/cast.py:1066: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  v = np.array(v, copy=False)
Out[2]:
[a, (a, b)]
Categories (2, object): [(a, b), a]

I think that perhaps the user should see this warning, but have an option to pass through a dtype to silence it... Not sure yet.

Tagged for backport to keep the CI passing on that branch.

[TomAugspurger]

I think right now I'd prefer the user take care of that categorical issue by passing in an ndarray. I'm going to update the tests to reflect that, but am certainly open to suggestions for how we can handle it within pandas.

Edit: 78514cf

So users will see the warning if they just pass Categorical(['a', ('a',)]), and the warning won't be especially clear since pandas is the one calling asarray, but I don't see a clean way to get the dtype argument through there.

[AlexKirko]

@TomAugspurger
From the data science standpoint, there is no meaning to a categorical variable having some kind of specific type. You wouldn't treat it any differently whether the values it can take are numbers, strings or tuples of DataFrames.
We could consider implementing a dtype=object as a default, either in this PR or in the future.
I'm not familiar with the constructor implementation though, so take this with a grain of salt.

[TomAugspurger]

We could consider implementing a dtype=object as a default, either in this PR or in the future.

We don't want that. You can have a categorical holding native types like integers, timestamps, etc. We want the .categories to continue to be backed by the appropriate index for that dtype

In [4]: type(pd.Categorical([1, 2]).categories)
Out[4]: pandas.core.indexes.numeric.Int64Index

[AlexKirko]

Seems fair. Especially as the warning is thrown in a pretty obscure use case, and changing the default would mess with all cases.

[jreback]

this looks good @TomAugspurger

In [2]: pd.Categorical(['a', ('a', 'b')])
/Users/taugspurger/sandbox/pandas/pandas/core/dtypes/cast.py:1066: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  v = np.array(v, copy=False)
Out[2]:
[a, (a, b)]
Categories (2, object): [(a, b), a]

agreed that this should just pass thru the warning (or maybe we want to have a warning; to having non-scalar based categories is odd); yes you can do it, but you should have to be specific about this

[TomAugspurger]

All green. Merging in ~1 hour to get CI passing again. I'll be able to do followups as needed.

[TomAugspurger]

FYI, at least in our tests, this isn't changing behavior. I added an assert com.index_labels_to_array(codes, dtype=object).dtype == com.index_labels_to_array(codes) temporarily, so we were always inferring object dtype here (in our tests).

[simonjayhawkins]

@TomAugspurger This needs a backport?

[TomAugspurger]

Yep. Not sure why the bot hasn't picked it up yet.

[TomAugspurger]

@meeseeksdev backport to 1.0.x

[jbrockmendel]

darn, i thought i had already fixed this one

[jbrockmendel]

this sort of changes the interface. do we want authors to handle this on their own?

[TomAugspurger]

How does it change the interface?

We are reducing coverage of _from_sequence inferring the dtype from an untyped list. We could restore that if desired (and probably skip for problematic arrays).

[jbrockmendel]

We are reducing coverage of _from_sequence inferring the dtype from an untyped list.

Yah, i guess that is a better description than "changes the interface"

[mattip]

Sorry for the breakage. and thanks for the quick fix. I thought I had tested this before re-merging the NumPy PR. Please ping me if I can help with any further issues.