BUG: CategoricalIndex.get_indexer issue with NaNs (pandas-dev#45361)

categorical_index_obj.get_indexer(target) yields incorrect results when
categorical_index_obj contains NaNs, and target does not. The reason
for this is that, if target contains elements which do not match any
category in categorical_index_obj, they are replaced by NaNs. In such
a situation, if categorical_index_obj also has NaNs, then the corresp
elements in target are mapped to an index which is not -1

eg:
ci = pd.CategoricalIndex([1, 2, np.nan, 3])
other = pd.Index([2, 3, 4])

ci.get_indexer(other)
In the implementation of get_indexer, other becomes [2, 3, NaN]
which is mapped to index 2, in ci

Author: Shashank-Shet

pandas/core/indexes/base.py

@@ -3696,6 +3696,10 @@ def get_indexer(
         tolerance=None,
     ) -> npt.NDArray[np.intp]:
         method = missing.clean_reindex_fill_method(method)
+        # After _maybe_cast_listlike_indexer, target elements which do not
+        # belong to some category are changed to NaNs
+        # Mask to track actual NaN values compared to inserted NaN values
+        target_nans = np.isnan(target)
         target = self._maybe_cast_listlike_indexer(target)
 
         self._check_indexing_method(method, limit, tolerance)
@@ -3720,7 +3724,8 @@ def get_indexer(
             if self.hasnans and target.hasnans:
                 loc = self.get_loc(np.nan)
                 mask = target.isna()
-                indexer[mask] = loc
+                indexer[target_nans] = loc
+                indexer[mask & ~target_nans] = -1
             return indexer
 
         if is_categorical_dtype(target.dtype):