Error Indexing DafaFrame with a 0-d array

[floatn]

>>> df = pd.DataFrame([[1, 2], [3, 4]], columns=['a', 'b'])
>>> ar = np.array(0)
>>> df.iloc[ar]
Traceback (most recent call last):
  ...
  ...
  File "/home/float2/Documents/Python/Pandas/pandas/pandas/core/indexes/base.py", line 529, in _shallow_copy_with_infer
    if not len(values) and 'dtype' not in kwargs:
TypeError: object of type 'numpy.int64' has no len()

A 0-d array passes this check cause it is an Iterable. But later, when len() is called on, TypeError is raised.
It would be nice to implement PEP 357 to prevent this error.
The matter has been raised at PyTorch project. #9237

[gfyoung]

cc @jreback

[chris-b1]

FYI #21861 brought this closer - is_list_like no longer returns true for 0d arrays, but still errors. I think unpacking the scalar would be the right thing to do here, as strange as 0d arrays are.

In [3]: >>> df.iloc[ar]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-9eb33c655c7c> in <module>()
----> 1 df.iloc[ar]

~\Documents\python-dev\pandas\pandas\core\indexing.py in __getitem__(self, key)
   1508
   1509             maybe_callable = com._apply_if_callable(key, self.obj)
-> 1510             return self._getitem_axis(maybe_callable, axis=axis)
   1511
   1512     def _is_scalar_access(self, key):

~\Documents\python-dev\pandas\pandas\core\indexing.py in _getitem_axis(self, key, axis)
   2216         # a single integer
   2217         else:
-> 2218             key = self._convert_scalar_indexer(key, axis)
   2219
   2220             if not is_integer(key):

~\Documents\python-dev\pandas\pandas\core\indexing.py in _convert_scalar_indexer(self, key, axis)
    264         ax = self.obj._get_axis(min(axis, self.ndim - 1))
    265         # a scalar
--> 266         return ax._convert_scalar_indexer(key, kind=self.name)
    267
    268     def _convert_slice_indexer(self, key, axis):

~\Documents\python-dev\pandas\pandas\core\indexes\numeric.py in _convert_scalar_indexer(self, key, kind)
    185             key = self._maybe_cast_indexer(key)
    186         return (super(Int64Index, self)
--> 187                 ._convert_scalar_indexer(key, kind=kind))
    188
    189     def _wrap_joined_index(self, joined, other):

~\Documents\python-dev\pandas\pandas\core\indexes\base.py in _convert_scalar_indexer(self, key, kind)
   1540
   1541         if kind == 'iloc':
-> 1542             return self._validate_indexer('positional', key, kind)
   1543
   1544         if len(self) and not isinstance(self, ABCMultiIndex,):

~\Documents\python-dev\pandas\pandas\core\indexes\base.py in _validate_indexer(self, form, key, kind)
   4158             pass
   4159         elif kind in ['iloc', 'getitem']:
-> 4160             self._invalid_indexer(form, key)
   4161         return key
   4162

~\Documents\python-dev\pandas\pandas\core\indexes\base.py in _invalid_indexer(self, form, key)
   1751                         "indexers [{key}] of {kind}".format(
   1752                             form=form, klass=type(self), key=key,
-> 1753                             kind=type(key)))
   1754
   1755     def get_duplicates(self):

TypeError: cannot do positional indexing on <class 'pandas.core.indexes.range.RangeIndex'> with these indexers [0] of <class 'numpy.ndarray'>

[floatn]

I don't think #21861 is a good fix here.

 return (isinstance(obj, Iterable) and
            # we do not count strings/unicode/bytes as list-like
            not isinstance(obj, string_and_binary_types) and
            # exclude zero-dimensional numpy arrays, effectively scalars
            not (isinstance(obj, np.ndarray) and obj.ndim == 0))

The last line looks like a bad decision.
A 0-d np.ndarray doesn't pass is_list_like anymore, but take, for example, a pytorch 0-d tensor:

>>> import torch
>>> tn = torch.tensor(0)
>>> df.iloc[tn]
Traceback (most recent call last):
...
...
  File "/home/float2/Documents/Python/Pandas/pandas/pandas/core/indexes/base.py", line 528, in _shallow_copy_with_infer
    if not len(values) and 'dtype' not in kwargs:
TypeError: object of type 'numpy.int64' has no len()

The Error hasn't changed and now, after #21861, two issues of the same nature has different output errors.

[chris-b1]

cc @jbrockmendel

@floatn, welcome a PR if you have a suggestion on how we should handle this, 0d arrays are an odd case by themselves, along with the complication of duck array types.

[jbrockmendel]

@floatn what behavior do you think makes sense with a zero-dim array?

Does torch.tensor subclass ndarray or otherwise present "standard" attributes that we can use to tell we are looking at a pseudo-scalar?

[floatn]

@jbrockmendel
I suggest using __index__() method. Try to call it on key within _validate_indexer. This may override is_integer(key) check there too, I guess. And add len() check to is_list_like_indexer to prevent 0-d iterables passing through. Something like that.

[illegalnumbers]

Is anyone working on this? I'd love to make a contribution if it's as simple as what you discussed.

[gfyoung]

@illegalnumbers : I don't think so. Go for it!

[illegalnumbers]

Cool I'm gonna start working on this now and hopefully have a PR today.

[illegalnumbers]

It seems that __index__() being called might not be the right answer since it raises a TypeError in the case that it is a correct shaped array object. It might be more beneficial to do something else otherwise I have to handle that exception when it is correct which seems wrong. Any other ideas?

[gfyoung]

There are two ways to look at this:

• Perhaps a special-case conditional to avoid the Exception?
• Follow the Zen of Python: It is better to ask for forgiveness than permission, in which case, Exception handling is fine 🙂

Your call!

[illegalnumbers]

Calling .ndim seems more appropriate since it doesn't error but I'm not sure if there is more to ___index___ that I might be missing context around?