BUG: fix calling local references with keyword arguments in query #26426

danielhrisca · 2019-05-16T11:50:44Z

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

calling local references with keyword arguments in query fails with attribute error:

import numpy as np
import pandas as pd

def f(a, b):
    return b

df = pd.DataFrame()

df['val'] = np.arange(10)
print(df.query('val > @f(4, 5)'))
print(df.query('val > @f(4, b=5)'))

   val
6    6
7    7
8    8
9    9
Traceback (most recent call last):

  File "C:/Users/uidn3651/Desktop/untitled0.py", line 10, in <module>
    print(df.query('val > @f(4, b=5)'))

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\frame.py", line 3052, in query
    res = self.eval(expr, **kwargs)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\frame.py", line 3169, in eval
    return _eval(expr, inplace=inplace, **kwargs)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\eval.py", line 293, in eval
    truediv=truediv)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 749, in __init__
    self.terms = self.parse()

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 766, in parse
    return self._visitor.visit(self.expr)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 370, in visit
    return visitor(node, **kwargs)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 376, in visit_Module
    return self.visit(expr, **kwargs)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 370, in visit
    return visitor(node, **kwargs)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 379, in visit_Expr
    return self.visit(node.value, **kwargs)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 370, in visit
    return visitor(node, **kwargs)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 674, in visit_Compare
    return self.visit(binop)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 370, in visit
    return visitor(node, **kwargs)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 476, in visit_BinOp
    op, op_class, left, right = self._maybe_transform_eq_ne(node)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 413, in _maybe_transform_eq_ne
    right = self.visit(node.right, side='right')

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 370, in visit
    return visitor(node, **kwargs)

  File "d:\work\02__pythonworkspace\_venv\lib\site-packages\pandas\core\computation\expr.py", line 657, in visit_Call
    kwargs.append(ast.keyword(

AttributeError: 'dict' object has no attribute 'append'

With the fix this works correctly

jreback

tests always are first

codecov · 2019-05-16T12:22:08Z

Codecov Report

Merging #26426 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #26426      +/-   ##
==========================================
- Coverage   91.74%   91.73%   -0.01%     
==========================================
  Files         174      174              
  Lines       50748    50748              
==========================================
- Hits        46557    46556       -1     
- Misses       4191     4192       +1

Flag	Coverage Δ
#multiple	`90.25% <100%> (+0.01%)`	⬆️
#single	`41.69% <0%> (-0.09%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/computation/expr.py	`97.52% <100%> (+0.82%)`	⬆️
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/frame.py	`97.02% <0%> (-0.12%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f44ac06...c7d58af. Read the comment docs.

danielhrisca · 2019-05-16T12:26:14Z

@jreback

running test_fast.bat produces many failed tests. However I can't find test_multi_line_expression_callable_local_variable in the output. Does this mean that this test is passed?

danielhrisca · 2019-05-17T19:35:50Z

@jreback
can you have a look at the new commit? Thank you

gfyoung · 2019-05-18T01:03:10Z

pandas/tests/computation/test_eval.py

+        """, inplace=True)
+        assert_frame_equal(expected, df)
+        assert ans is None
+


Split this test into two.

@gfyoung
done

jreback

add a whatsnew note; reshaping or other section would be ok

pandas/tests/computation/test_eval.py

doc/source/whatsnew/v0.25.0.rst

jreback · 2019-05-19T18:42:52Z

you will need to merge master as well, there is a conflict

…yword_in_query

danielhrisca · 2019-05-19T18:51:09Z

Th conflict is now resolved

jreback

need a different format for the issue number

jreback · 2019-05-19T18:53:29Z

doc/source/whatsnew/v0.25.0.rst

@@ -380,6 +379,7 @@ Indexing
 - Bug in which :meth:`DataFrame.append` produced an erroneous warning indicating that a ``KeyError`` will be thrown in the future when the data to be appended contains new columns (:issue:`22252`).
 - Bug in which :meth:`DataFrame.to_csv` caused a segfault for a reindexed data frame, when the indices were single-level :class:`MultiIndex` (:issue:`26303`).
 - Fixed bug where assigning a :class:`arrays.PandasArray` to a :class:`pandas.core.frame.DataFrame` would raise error (:issue:`26390`)
+- Allow keyword arguments for callable local reference used in the :method:`DataFrame.query` string (PR 26426)


this needs to be like the other, e.g. (:issue:`26426`)

Done. Thank you for the patience!

jreback · 2019-05-19T20:27:10Z

thanks for the patch @danielhrisca

fix calling local references with keyword arguments in query

c492d96

jreback requested changes May 16, 2019

View reviewed changes

danielhrisca changed the title ~~fix calling local references with keyword arguments in query~~ BUG: fix calling local references with keyword arguments in query May 16, 2019

add test

634dc06

gfyoung added the Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff label May 18, 2019

gfyoung reviewed May 18, 2019

View reviewed changes

split test into separate functions

49dae52

jreback requested changes May 18, 2019

View reviewed changes

pandas/tests/computation/test_eval.py Show resolved Hide resolved

pandas/tests/computation/test_eval.py Show resolved Hide resolved

add whatsnew entry

9e497df

jreback requested changes May 19, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Outdated Show resolved Hide resolved

jreback mentioned this pull request May 19, 2019

BUG: Fix bug when using df.query with "str.contains()" #25813

Closed

3 tasks

Update v0.25.0.rst

169729a

jreback added this to the 0.25.0 milestone May 19, 2019

danielhrisca added 2 commits May 19, 2019 21:47

Merge branch 'master' of https://github.com/pandas-dev/pandas into ke…

1d39377

…yword_in_query

add PR reference in the tests

b31ff8b

jreback requested changes May 19, 2019

View reviewed changes

Update v0.25.0.rst

c7d58af

jreback approved these changes May 19, 2019

View reviewed changes

jreback merged commit e4f727b into pandas-dev:master May 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: fix calling local references with keyword arguments in query #26426

BUG: fix calling local references with keyword arguments in query #26426

danielhrisca commented May 16, 2019 •

edited

Loading

jreback left a comment

codecov bot commented May 16, 2019 •

edited

Loading

danielhrisca commented May 16, 2019

danielhrisca commented May 17, 2019

gfyoung May 18, 2019

danielhrisca May 18, 2019

jreback left a comment

jreback commented May 19, 2019

danielhrisca commented May 19, 2019

jreback left a comment

jreback May 19, 2019

danielhrisca May 19, 2019

jreback commented May 19, 2019

BUG: fix calling local references with keyword arguments in query #26426

BUG: fix calling local references with keyword arguments in query #26426

Conversation

danielhrisca commented May 16, 2019 • edited Loading

jreback left a comment

Choose a reason for hiding this comment

codecov bot commented May 16, 2019 • edited Loading

Codecov Report

danielhrisca commented May 16, 2019

danielhrisca commented May 17, 2019

gfyoung May 18, 2019

Choose a reason for hiding this comment

danielhrisca May 18, 2019

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback commented May 19, 2019

danielhrisca commented May 19, 2019

jreback left a comment

Choose a reason for hiding this comment

jreback May 19, 2019

Choose a reason for hiding this comment

danielhrisca May 19, 2019

Choose a reason for hiding this comment

jreback commented May 19, 2019

danielhrisca commented May 16, 2019 •

edited

Loading

codecov bot commented May 16, 2019 •

edited

Loading