BUG: CategoricalIndex allows reindexing with non-unique CategoricalIndex

[qwhelan]

CategoricalIndex.reindex() throws a ValueError if the target index is non-unique and not also a CategoricalIndex. This means a non-unique CategoricalIndex can be used to reindex, resulting in duplicated data:

In [1]: import pandas as pd
        categorical_index = pd.CategoricalIndex(['bar', 'foo', 'bar', 'foo'], ['bar', 'foo'])
        s = pd.Series(range(4), index=categorical_index)

In [2]: s
Out[2]: bar    0
        foo    1
        bar    2
        foo    3
        dtype: int64

In [3]: s.reindex(categorical_index.as_ordered())
Out[3]: bar    0
        bar    2
        foo    1
        foo    3
        bar    0
        bar    2
        foo    1
        foo    3
        dtype: int64

This PR removes the extraneous type check and adds tests checking that an exception is thrown for non-unique target indexes, regardless of type.

• closes CategoricalIndex reidex duplicates values #21809
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[pep8speaks]

Hello @qwhelan! Thanks for updating the PR.

• There are no PEP8 issues in the file pandas/core/indexes/category.py !

• There are no PEP8 issues in the file pandas/tests/frame/test_analytics.py !

• There are no PEP8 issues in the file pandas/tests/indexes/test_category.py !

Comment last updated on November 28, 2018 at 22:26 Hours UTC

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@6dd130a). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master   #23963   +/-   ##
=========================================
  Coverage          ?    92.3%           
=========================================
  Files             ?      161           
  Lines             ?    51558           
  Branches          ?        0           
=========================================
  Hits              ?    47592           
  Misses            ?     3966           
  Partials          ?        0

Flag	Coverage Δ
#multiple	90.7% <100%> (?)
#single	42.43% <57.14%> (?)

Impacted Files	Coverage Δ
pandas/core/indexes/category.py	97.88% <100%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6dd130a...9550880. Read the comment docs.

[jreback]

can you a whatsnew entry, and replicate the df.round() (that should work now) in the OP

[qwhelan]

@jreback I stumbled across this bug while writing some asv benchmarks and only noticed #21809 after opening this PR, so it ended up being a few more lines to handle the df.round() case.

As we're now raising on non-unique CategoricalIndexes, we need to whitelist non-unique targets that are also the current index to allow df.round() to run. As the construction of the result involves a .reindex() call, we'd otherwise get ValueErrors when calling df.round() on a non-uniquely indexed DataFrame.

This is already done in Index.reindex(), so this whitelisting is bringing CategoricalIndex back in line with the base implementation.

[qwhelan]

Additionally, this is the fixed output from #21809:

print(dfb.round(3))
         a      b      c
low  0.087  0.230  0.411
low  0.311  0.566  0.545
low  0.807  0.918  0.522
hi   0.425  0.072  0.899
hi   0.421  0.582  0.214
hi   0.447  0.468  0.101

[jreback]

can you construct the result and use tm.assert_frame_equal here

[qwhelan]

Added a tm.assert_frame_equal() check but keeping the explicit dims check as I don't want to inadvertently rely on CategoricalIndex.reindex.

[qwhelan]

@jreback Changes implemented, the one failing test is a flaky hypothesis failure in an unrelated file.

[jreback]

cosmestic change in tests. ping when green.

[qwhelan]

@jreback Green other than the unrelated flaky test

[jreback]

thanks @qwhelan