ENH: support Ellipsis in loc/iloc

[jbrockmendel]

• closes #xxxx
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

closes #10956

[WillAyd]

I realize there is still an open issue for this but it is quite old. What do you see as the main motivation to support this given we no longer have a Panel?

[jbrockmendel]

What do you see as the main motivation to support this given we no longer have a Panel?

As @shoyer mentioned in #10956, this makes it easier to write code that is valid for both Series and DataFrame, e.g. obj.loc[..., labels]

In particular Im finding this would be useful for parametrizing indexing tests.

[jreback]

I agree this is good for general indexing compatiblity and is unambiguous.

@jbrockmendel can you make this raise for a MI or is it unambiguous there? (I think it is more useful than the slice syntax, but that should be a dedicated PR).

cc @jorisvandenbossche

[jbrockmendel]

can you make this raise for a MI or is it unambiguous there? (I think it is more useful than the slice syntax, but that should be a dedicated PR).

MI cases are appreciably more complicated and im planning to handle them separately

[jorisvandenbossche]

I personally don't see myself using it, but the arguments of @shoyer on the issue certainly make sense (although they are from a time we still had multidimensional dataframes (>2d) I think)

Now, if adding it, this will need some docs.

Also, I think we would need some some more test cases?
Eg df.loc[..., ...] (in numpy this errors about "an index can only have a single ellipsis ('...')")
Or series.loc[...], or df.loc[..., 1, 2], ...

In numpy, arr[.., 1] and arr[1] actually do something different on a 1D array (numpy scalar vs 0d array; now we don't have this difference in pandas, so that's probably not an issue)

MI cases are appreciably more complicated and im planning to handle them separately

Do you mean having an Ellipsis "within" the indexer for the MultiIndexes axis?
That's indeed certainly a different topic, but just having an Ellipsis when indexing a dataframe with a MultiIndex can be tested here, I think, as it is already covered by your changes (I assume). Eg the same tests as you have now, like df.loc[..., [1]] but on a DataFrame with a MI (I suppose this doesn't take a different code path necessarily, as it just expands to a null slice, but still good to explicitly have tests with different index types I think)

[jorisvandenbossche]

Can you rename this method if it now actually returns a modified key?

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[jreback]

i think this is ok, we should be able to accept ellipsis (even if not that useful). add a whatsnew and rebase. ping on green (i think @jorisvandenbossche had a comment as well).

[jreback]

@jbrockmendel can you rebase this

[jbrockmendel]

can you rebase this

sure, but there are still a handful of comments I havent addressed, so this isnt ready

[jbrockmendel]

updated to flesh out tests, i think comments are addressed

[simonjayhawkins]

@jbrockmendel if this is ready can you add a release note and a section on the new capability.

[jbrockmendel]

whatsnew added + green

[simonjayhawkins]

@jbrockmendel needs rebase

[jbrockmendel]

rebased + green

[jreback]

it would be a nice extension to support ... in both dimensions as we do for : today (followon)
(and maybe in multi-index) but i am not sure it adds a lot of value.

Actually i dont' think this is any different than : right ultimately right?

[jbrockmendel]

Actually i dont' think this is any different than : right ultimately right?

The main thing this allows is to write code that works for either Series or DataFrame obj.loc[..., "foo"]. IIRC the original motivation was to make sharing tests easier.

[jreback]

Actually i dont' think this is any different than : right ultimately right?

The main thing this allows is to write code that works for either Series or DataFrame obj.loc[..., "foo"]. IIRC the original motivation was to make sharing tests easier.

ahh got it. then am +1 here.

cc @phofl if any comments.

[jreback]

can you create an issue for adding docs about this in the user guide / doc-string as appropriate.

[phofl]

This looks good.

Is
df = pd.DataFrame({"a": [1, 2, 3]}).loc[(..., ), :] valid syntax? cause this raises a KeyError

[jbrockmendel]

df = pd.DataFrame({"a": [1, 2, 3]}).loc[(..., ), :] valid syntax?

If you mean "should it be equivalent to .loc[..., :]?" then no. But...

index = [0, Ellipsis, NotImplemented]
df = pd.DataFrame({"a": [1, 2, 3]}, index=index)

>>> df.loc[(..., ), :]
          a
Ellipsis  2

(on master, not PR branch)

[jreback]

thanks @jbrockmendel