Skip to content

BUG: UndefinedVariableError in DataFrame.query if name is type #48694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3 tasks done
pm13 opened this issue Sep 21, 2022 · 2 comments
Open
3 tasks done

BUG: UndefinedVariableError in DataFrame.query if name is type #48694

pm13 opened this issue Sep 21, 2022 · 2 comments
Assignees
Labels
expressions pd.eval, query Regression Functionality that used to work in a prior pandas version

Comments

@pm13
Copy link

pm13 commented Sep 21, 2022

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df = pd.DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})

class A:
    a = 1

df.query('x == @A.a')  # <-- this produces the error

Issue Description

In Pandas 1.5.0 it is not possible to use class as name in DataFrame.query:

>>> import pandas as pd
>>> 
>>> df = pd.DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})
>>> 
>>> class A:
...     a = 1
... 
>>> df.query('x == @A.a')  # <-- this produces the error
Traceback (most recent call last):
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/scope.py", line 198, in resolve
    return self.resolvers[key]
  File "/usr/lib64/python3.10/collections/__init__.py", line 986, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/usr/lib64/python3.10/collections/__init__.py", line 978, in __missing__
    raise KeyError(key)
KeyError: 'A'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/scope.py", line 209, in resolve
    return self.temps[key]
KeyError: 'A'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/util/_decorators.py", line 317, in wrapper
    return func(*args, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/frame.py", line 4464, in query
    res = self.eval(expr, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/util/_decorators.py", line 317, in wrapper
    return func(*args, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/frame.py", line 4602, in eval
    return _eval(expr, inplace=inplace, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/eval.py", line 354, in eval
    parsed_expr = Expr(expr, engine=engine, parser=parser, env=env)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 813, in __init__
    self.terms = self.parse()
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 832, in parse
    return self._visitor.visit(self.expr)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 421, in visit_Module
    return self.visit(expr, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 424, in visit_Expr
    return self.visit(node.value, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 723, in visit_Compare
    return self.visit(binop)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 536, in visit_BinOp
    op, op_class, left, right = self._maybe_transform_eq_ne(node)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 458, in _maybe_transform_eq_ne
    right = self.visit(node.right, side="right")
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 645, in visit_Attribute
    resolved = self.visit(value).value
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/expr.py", line 549, in visit_Name
    return self.term_type(node.id, self.env, **kwargs)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/ops.py", line 85, in __init__
    self._value = self._resolve_name()
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/ops.py", line 109, in _resolve_name
    res = self.env.resolve(local_name, is_local=is_local)
  File "/home/petr/.local/venv/lib64/python3.10/site-packages/pandas/core/computation/scope.py", line 211, in resolve
    raise UndefinedVariableError(key, is_local) from err
pandas.errors.UndefinedVariableError: name 'A' is not defined

I think it is after this commit:
8b36142

I can get expected behavior if I comment this condition:
https://github.com/pandas-dev/pandas/blob/main/pandas/core/computation/ops.py#L105

Expected Behavior

>>> import pandas as pd
>>> 
>>> df = pd.DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})
>>> 
>>> class A:
...     a = 1
... 
>>> df.query('x == @A.a')  # <-- this produces the error
   x  y
0  1  4

Installed Versions

INSTALLED VERSIONS

commit : 87cfe4e
python : 3.10.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.19.9-200.fc36.x86_64
Version : #1 SMP PREEMPT_DYNAMIC Thu Sep 15 09:49:52 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.5.0
numpy : 1.23.3
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 65.3.0
pip : 22.2.2
Cython : None
pytest : 7.1.3
hypothesis : None
sphinx : 5.1.1
blosc : None
feather : None
xlsxwriter : 3.0.3
lxml.etree : 4.9.1
html5lib : 1.1
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.5.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : 1.3.5
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.6.0
numba : None
numexpr : 2.8.3
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.1
snappy : None
sqlalchemy : 1.4.41
tables : None
tabulate : 0.8.10
xarray : None
xlrd : 2.0.1
xlwt : None
zstandard : None
tzdata : None

@pm13 pm13 added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 21, 2022
@phofl
Copy link
Member

phofl commented Sep 21, 2022

Can confirm

commit 8b36142a0e7c1071e9fd9ff37f579ebdef056838
Author: Khor Chean Wei <[email protected]>
Date:   Fri Aug 12 08:15:48 2022 +0800

    Error raised message: DataFrame eval will not work with 'Timestamp' column name (#47273)

cc @weikhor

@phofl phofl added this to the 1.5.1 milestone Sep 21, 2022
@phofl phofl added Regression Functionality that used to work in a prior pandas version expressions pd.eval, query and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 21, 2022
@rhshadrach
Copy link
Member

Somewhat tangential, but with f-strings, is there any advantage of using @? E.g. in this case

df.query(f'x == {A.a}')

This has the added benefit of linters detecting the usage of A.a.

@datapythonista datapythonista modified the milestones: 1.5.1, 1.5.2 Oct 20, 2022
@debnathshoham debnathshoham self-assigned this Oct 25, 2022
@datapythonista datapythonista modified the milestones: 1.5.2, 1.5.3 Nov 15, 2022
@datapythonista datapythonista modified the milestones: 1.5.3, 1.5.4 Jan 18, 2023
@datapythonista datapythonista modified the milestones: 1.5.4, 2.0 Feb 27, 2023
@MarcoGorelli MarcoGorelli modified the milestones: 2.0, 2.1 Mar 27, 2023
@lithomas1 lithomas1 removed this from the 2.1 milestone Aug 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
expressions pd.eval, query Regression Functionality that used to work in a prior pandas version
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants