BUG (string dtype): comparison of string column to mixed object column fails #60228 (fixed)

[TEARFEAR]

Changes :

• Updated comparison_op function in pandas/core/ops/array_ops.py:
  • Added handling for comparisons between string and object dtypes.
  • Ensured string and object types are casted to a common type (string) before comparison.
• Added a test case to validate the fix:
  • Verified comparison works for string vs object with homogeneous and mixed types.
  • Verified behavior with PyArrow-based strings enabled (pd.options.future.infer_string = True).

--

• closes BUG (string dtype): comparison of string column to mixed object column fails #60228 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[jorisvandenbossche]

Thanks for working on this!
Added a few suggestions

[jorisvandenbossche]

Checking for string dtype for an array can be expensive in case the array is object dtype (at that point it will scan all values to check if they are strings). So we might want to try avoid that at this level.
I think we could handle the issue specifically for the ArrowExtensionArray itself (see the code I referenced in #60228 (comment))

[jorisvandenbossche]

We might need to do the casting the other way around. Instead of casting the object to string and then compare both as strings, I think we have to cast the string to object and compare both as object dtype.

The reason for this is that casting to string might actually convert values to strings, and then we are no longer doing the comparison for the original values.

>>> ser_string = pd.Series(["1", "b"])
>>> ser_mixed = pd.Series([1, "b"])
>>> ser_string == ser_mixed
0    False
1     True
dtype: bool

>>> ser_string == ser_mixed.astype("string")
0    True
1    True
dtype: bool

So if we would do that casting under the hood, the result would change in this case.

[jorisvandenbossche]

And we should add this case to the tests!

[jorisvandenbossche]

You don't need to add this for CI, because we have a separate CI build that enables this option for the full test suite.

Now, this can still be useful to test locally, but the way you can do this is with setting an environment variable (on linux I can do PANDAS_FUTURE_INFER_STRING=1 pytest ... to run the test with the option enabled.

[jorisvandenbossche]

@TEARFEAR git workflow related comment: it seems you have picked up some unrelated commits. In general doing something like

git fetch upstream
git merge upstream/main
git push origin bug-update-60228

should fix that (assuming origin is your fork and upstream is the pandas-dev/pandas remote)

[jorisvandenbossche]

(small ping to see if you have some time to update this PR)

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[jorisvandenbossche]

@TEARFEAR would you have time to update this PR? (otherwise I am also happy to update it, as we would like to include this in the 2.3 release)

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[mroeschke]

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.