add Column.unique_indices

[MarcoGorelli]

closes #135

(related: I'll make a PR next week to unify keys and labels, there's some inconsistencies)

[kkraus14]

Should we note how null values are or aren't handled here? (and possibly NaN values?)

[rgommers]

Should we note how null values are or aren't handled here? (and possibly NaN values?)

For nan, how about copying the array API's unique semantics: https://data-apis.org/array-api/draft/API_specification/generated/array_api.unique_values.html? Keeps things consistent. I'll also note that:

• numpy.unique has an equal_nan=True keyword
• pandas.Series doesn't document what it's doing, but considers them equal

pandas.DataFrame doesn't have a unique method - do we need it here at the dataframe level?

For null values, should they simply be discarded? They're missing, and I'd expect unique to return a set of the non-missing data in the column.

[MarcoGorelli]

pandas.DataFrame doesn't have a unique method - do we need it here at the dataframe level?

TBH we can probably just keep it out for now

I'd keep null values, as a user I'd prefer to know if there are null values

[MarcoGorelli]

For nan, how about copying the array API's unique semantics: https://data-apis.org/array-api/draft/API_specification/generated/array_api.unique_values.html? Keeps things consistent

let's bring this up on the call

[rgommers]

For nan, how about copying the array API's unique semantics: https://data-apis.org/array-api/draft/API_specification/generated/array_api.unique_values.html? Keeps things consistent

let's bring this up on the call

Actually, this is also covered in #128 (comment). Let's follow that (say "it's implementation-specific, NaN position isn't guaranteed"). I feel like we've been over this a lot, and it keeps on coming up on various PRs. So let's move this along I suggest, and ignore my proposal to make it match the array standard.

Probably deserves a separate design topics page on NaN and null handling that can be linked to, and we use as a reference whenever this comes up? I can write that.

[jorisvandenbossche]

I'd keep null values, as a user I'd prefer to know if there are null values

Another option is to have a keyword for this (like reductions also have one, and in pandas groupby() and value_counts() also have a keyword for this).

A case where you might not want the null values: if you use unique to mimic iterating over groupby groups (which we decided to not support), getting the null wouldn't work in a subsequent loop subsetting the dataframe (df.get_rows_by_mask(df.get_column_by_name(col) == unique_val))

[kkraus14]

Should we consider making this unique_indices similar to what we did for sorted_indices? Is there ever a case where someone wants to know where they should go for the unique values as opposed to getting the output values directly?

It's cheap to go from the indices to the values, but expensive to go from the values to the indices.

[rgommers]

Some points we discussed today:

• We want to return a single nan only, not multiple. This is what existing dataframe libraries do. The array API standard choice for multiple nans was driven by performance (it's faster on GPU); array libraries are more flop-constrained than dataframes; for dataframes the extra computation should be fine. And it'd be less implementation work.
• We want to indeed return indices with unique_indices, not values with unique.
  • There are use cases for getting indices from col.unique() and then using that to retrieve rows from a dataframe; the non-determinism is not always a problem (e.g., the subset keyword of pd.DataFrame.value_counts) has this behavior).
• We're fine with no ordering is guaranteed, and the function is not necessarily even deterministic in case of duplicate value. No guarantees beyond "returns an index to each unique value in the column".

[rgommers]

LGTM too

[rgommers]

Looks good now, so in it went. Thanks Marco and Keith!