BUG: query on columns with characters like # in its name

[aram-cedarwood]

• closes BUG: Error on query function when the column name has # symbol #59285
• closes BUG: Cannot parse query for df.query() with column name containing two-bytes parenthesis. #49633
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

---

This PR first added parsing.py: #28215

---

The issue is that none of the libraries tokenize, ast, or numexpr handle # the way we want it to.

tokenize is used here:

pandas/pandas/core/computation/expr.py

Line 169 in 18a3eec

return tokenize.untokenize(f(x) for x in tokenize_string(source))

pandas/pandas/core/computation/parsing.py

Line 186 in 18a3eec

token_generator = tokenize.generate_tokens(line_reader)

ast is used here:

pandas/pandas/core/computation/expr.py

Line 407 in 18a3eec

node = ast.fix_missing_locations(ast.parse(clean))

numexpr is used here:

pandas/pandas/core/computation/engines.py

Line 129 in 18a3eec

return ne.evaluate(s, local_dict=scope)

[WillAyd]

Minor comments / questions but overall this looks great. @jreback in case you have more feedback on your end

[WillAyd]

I'm assuming these strings don't get too large, but would definitely be better for performance to collect each element you loop over into a list and join at the end, rather than using add assignment

[WillAyd]

@aram-cinnamon for the pre-commit issue you can see if you find anything of note in https://github.com/pre-commit/pygrep-hooks/issues, but I am doubtful that there is any real solve. You may just need to add exclude: ^pandas/tests/frame/test_query_eval.py to the rst-inline-touching-normal hook in the .pre-commit-config.yaml file

[aram-cedarwood]

Thanks so much Will!!

[aram-cedarwood]

Hm tests still failing. Will look into it soon

[WillAyd]

Hmm the failure on the Future Infer Strings job is kind of tricky, and tangential to what you are doing. For now I would just add the @pytest.mark.xfail(using_string_dtype(), reason="TODO(infer_string)") decorator that you see on some other tests

[WillAyd]

lgtm @mroeschke

[WillAyd]

Thanks @aram-cinnamon - very impressive!

[tgpaul-lm]

Great work! Had been hoping to see this in the next patch release which was 2.2.3 (released 20-sep-2024), but didn't make it. Any thoughts when this might make its way into an official release?

[WillAyd]

It will be part of the 3.0 release