ENH: clearer error msg for unequal categoricals in merge_asof (GH#26136)

[chrish42]

• closes Confusing "MergeError: incompatible merge keys [1] category and category, must be the same type" #26136
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[WillAyd]

Do we have a test for this that ensures this works when they are "equal"?

[codecov]

Codecov Report

Merging #26242 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #26242      +/-   ##
==========================================
- Coverage   91.97%   91.96%   -0.01%     
==========================================
  Files         175      175              
  Lines       52368    52371       +3     
==========================================
  Hits        48164    48164              
- Misses       4204     4207       +3

Flag	Coverage Δ
#multiple	90.52% <100%> (ø)	⬆️
#single	40.69% <0%> (-0.16%)	⬇️

Impacted Files	Coverage Δ
pandas/core/reshape/merge.py	94.49% <100%> (+0.02%)	⬆️
pandas/io/gbq.py	78.94% <0%> (-10.53%)	⬇️
pandas/core/frame.py	96.9% <0%> (-0.12%)	⬇️
pandas/util/testing.py	90.71% <0%> (+0.1%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9feb3ad...f89c29c. Read the comment docs.

[codecov]

Codecov Report

Merging #26242 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #26242      +/-   ##
==========================================
+ Coverage   91.97%   91.97%   +<.01%     
==========================================
  Files         175      175              
  Lines       52368    52389      +21     
==========================================
+ Hits        48164    48184      +20     
- Misses       4204     4205       +1

Flag	Coverage Δ
#multiple	90.52% <100%> (ø)	⬆️
#single	40.72% <0%> (-0.13%)	⬇️

Impacted Files	Coverage Δ
pandas/core/reshape/merge.py	94.49% <100%> (+0.02%)	⬆️
pandas/io/gbq.py	78.94% <0%> (-10.53%)	⬇️
pandas/io/excel/_base.py	92.13% <0%> (-0.72%)	⬇️
pandas/core/indexes/datetimelike.py	98.13% <0%> (-0.4%)	⬇️
pandas/core/base.py	97.98% <0%> (-0.22%)	⬇️
pandas/plotting/_core.py	83.63% <0%> (-0.14%)	⬇️
pandas/core/frame.py	96.9% <0%> (-0.12%)	⬇️
pandas/core/generic.py	93.54% <0%> (ø)	⬆️
pandas/core/arrays/period.py	98.53% <0%> (ø)	⬆️
pandas/core/arrays/interval.py	93.06% <0%> (ø)	⬆️
... and 10 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9feb3ad...464fa34. Read the comment docs.

[chrish42]

Good question, actually. Yes, there is. It's test_basic_categorical (in the same file). But before I added the new test in this pull request, that was the only test that exercised the code paths of merge_asof for categoricals. I feel there should be more of those... That would be another pull request IMHO. though.

[WillAyd]

Thanks for clarifying. So this change does introduce unordered Categoricals which might be counter to what merge_asof is supposed to offer (the docstring explicitly states the join field must be ordered).

Do you you see any other instances of that method supporting unordered data? Wondering if we need to further require that Categoricals are ordered to use this method

[chrish42]

@WillAyd Indeed, the merge keys should be ordered, otherwise merge_asof() makes no sense. Currently, the merge fails later (in _get_join_indexers()) with a ValueError. But fun fact, there is no test currently in test_merge_asof.py. I've added a test that undordered categorical merge keys raises.

I've also added code to raise a MergeError (instead of a ValueError further down the road). But maybe adding this error check in in _get_merge_keys() is not the right place, because by then, the by= keys have been added to the list of merge keys.

Is there a reliable way to skip over the by= keys in this new test? Or should I put this test in another method? Thanks!

[WillAyd]

Hmm that's unfortunate. If it makes things too complicated I'm not against looking that in more detail as a follow up if you can open an issue for it

[chrish42]

Cool. I've reverted the last change, and added a comment to explain what's happening currently. I've created #26249 for the other cleanups.

[WillAyd]

Thanks for opening an issue for follow ups. I'm OK with this as is with cleanups to follow. Over to @jreback

[jreback]

thanks @chrish42 nice patch!