REGR: comparison op with dask data structure fails

[jorisvandenbossche]

Using pandas 1.0.5 and latest dask 2020.12.0:

In [2]: import dask.dataframe as dd                                                                                                                                                                                

In [3]: df = pd.DataFrame({"x": ["a", "b", "c"] * 100}, dtype="category") 
   ...: ddf = dd.from_pandas(df, npartitions=3)                                                                                                                                                                

In [4]: df.x                                                                                                                                                                                                       
Out[4]: 
0      a
1      b
2      c
3      a
4      b
      ..
295    b
296    c
297    a
298    b
299    c
Name: x, Length: 300, dtype: category
Categories (3, object): [a, b, c]

In [5]: ddf.x                                                                                                                                                                                                      
Out[5]: 
Dask Series Structure:
npartitions=3
0      category[known]
100                ...
200                ...
299                ...
Name: x, dtype: category
Dask Name: getitem, 6 tasks

In [6]: df.x == ddf.x                                                                                                                                                                                              
Out[6]: 
0      True
1      True
2      True
3      True
4      True
       ... 
295    True
296    True
297    True
298    True
299    True
Name: x, Length: 300, dtype: bool

In [9]: (df.x == ddf.x).all()                                                                                                                                                                                      
Out[9]: True

But with master (using same dask version), this gives:

In [3]: df.x == ddf.x
Out[3]: 
0      False
1      False
2      False
3      False
4      False
       ...  
295    False
296    False
297    False
298    False
299    False
Name: x, Length: 300, dtype: bool

[jorisvandenbossche]

Checked and this was still working in 1.1.x, and started to fail for pandas 1.2.0

[jorisvandenbossche]

Bisecting gives

40daf00 is the first bad commit
commit 40daf00
Author: jbrockmendel [email protected]
Date: Fri Oct 2 15:43:23 2020 -0700

BUG: Categorical setitem, comparison with tuple category (#36623)

cc @jbrockmendel

[jorisvandenbossche]

Hmm, it seems that this before only worked for categorical (at least testing the above example but using int64 or object dtype, I get all False or an error, respectively, also on previous released versions of pandas). So probably not a priority for 1.2.x

[jbrockmendel]

Looks like ddf is hashable so we treat it as scalar-like. I think deep down the thing to do is return NotImplemented and let dask handle alignment

[simonjayhawkins]

moved to 1.2.2

[simonjayhawkins]

moved to 1.2.3

[jbrockmendel]

@TomAugspurger suggestions for what attribute to check to determine we should return NotImplemented?

[simonjayhawkins]

moved to 1.2.5

[simonjayhawkins]

So probably not a priority for 1.2.x

removing milestone.

[jbrockmendel]

Following #48347, it is up to dask to define __pandas_priority__ to fix this