Cannot parse query for df.query() with column name containing parenthesis.

[ash-ishh]

Code:

 import pandas as pd

 a = {'abc':[1,2,3],'xy (z)':[4,5,6]}

 df = pd.DataFrame(a)
 query = "`xy (z)` == 2"
 new_df = df.query(query)

Error :

xy_(z)_BACKTICK_QUOTED_STRING ==2
SyntaxError: invalid syntax

Problem description

df.query is not able to parse the query where column name of df is not valid python identifier.

Workaround: https://stackoverflow.com/questions/40045545/pandas-query-string-where-column-name-contains-special-characters

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.6.8.final.0
python-bits : 64
OS : Linux
OS-release : 4.4.187-1.el7.elrepo.x86_64
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.1
numpy : 1.17.0
pytz : 2019.2
dateutil : 2.8.0
pip : 18.1
setuptools : 40.6.2
Cython : 0.28.2
pytest : 5.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.0.4
lxml.etree : 4.4.1
html5lib : 1.0.1
pymysql : 0.9.3
psycopg2 : 2.8.3 (dt dec pq3 ext lo64)
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : 4.8.0
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.4.1
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 2.6.2
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.1.0
sqlalchemy : 1.3.3
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : 1.0.4

[dhwanils95]

Yess, I am facing the exact same issue. The column name has paranthesis character which it is not able to parse and is raising exception.

[TomAugspurger]

I think we have an open PR allowing non-valid python identifiers: #28215 You could check if it covers this case.

…

On Mon, Sep 23, 2019 at 2:27 AM Dhwanil Shah ***@***.***> wrote: Yess, I am facing the exact same issue. The column name has paranthesis character which it is not able to parse and is raising exception. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#28576?email_source=notifications&email_token=AAKAOIRGKQA6IPFUGI6GPHDQLBVWLA5CNFSM4IZHESIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7KACXQ#issuecomment-533987678>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOISJPY5UDDYFUDXDOHDQLBVWLANCNFSM4IZHESIA> .

[ash-ishh]

I tried above commit but the case doesn't seem to work.
Also the functionality mentioned in PR (digit, start with a number, or are separated by operators) is not working.

I installed pandas from this commit :
commit 179cd3b
Merge: 08b6400 23e7595
Author: Hielke Walinga [email protected]
Date: Thu Sep 12 16:10:02 2019 +0200
Merge branch 'allow-special-characters-query' of https://github.com/hwalinga/pandas into allow-special-characters-query

Pandas version: 0.24.0.dev0+2984.g179cd3b

[hwalinga]

Hello, thanks for the issue.

This is indeed an edge case. I have this working now. Check the latest commits.

[ash-ishh]

Thank you @hwalinga it is working now :)