BUG: Fix bug when using df.query with "str.contains()" #25813

dtpc · 2019-03-21T01:36:14Z

tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Fixes bug in on line 588 of pandas/core/computation/expr.py: AttributeError: 'dict' object has no attribute 'append'

Also addresses the specific example in #22172

Example:

import pandas as pd
df = pd.DataFrame([["I", "XYZ"], ["IJ", None]], columns=['A', 'B'])
df.query("B.str.contains('Z', na=False)", engine="python")

Traceback (most recent call last):
  File "./test_query.py", line 12, in <module>
    result = df.query("B.str.contains('Z', na=False)", engine="python")
  File "./lib/python3.6/site-packages/pandas/core/frame.py", line 2847, in query
    res = self.eval(expr, **kwargs)
  File "./lib/python3.6/site-packages/pandas/core/frame.py", line 2962, in eval
    return _eval(expr, inplace=inplace, **kwargs)
  File "./lib/python3.6/site-packages/pandas/core/computation/eval.py", line 291, in eval
    truediv=truediv)
  File "./lib/python3.6/site-packages/pandas/core/computation/expr.py", line 739, in __init__
    self.terms = self.parse()
  File "./lib/python3.6/site-packages/pandas/core/computation/expr.py", line 756, in parse
    return self._visitor.visit(self.expr)
  File "./lib/python3.6/site-packages/pandas/core/computation/expr.py", line 321, in visit
    return visitor(node, **kwargs)
  File "./lib/python3.6/site-packages/pandas/core/computation/expr.py", line 327, in visit_Module
    return self.visit(expr, **kwargs)
  File "./lib/python3.6/site-packages/pandas/core/computation/expr.py", line 321, in visit
    return visitor(node, **kwargs)
  File "./lib/python3.6/site-packages/pandas/core/computation/expr.py", line 330, in visit_Expr
    return self.visit(node.value, **kwargs)
  File "./lib/python3.6/site-packages/pandas/core/computation/expr.py", line 321, in visit
    return visitor(node, **kwargs)
  File "./lib/python3.6/site-packages/pandas/core/computation/expr.py", line 588, in visit_Call_35
    kwargs.append(ast.keyword(
AttributeError: 'dict' object has no attribute 'append'

codecov · 2019-03-21T02:14:05Z

Codecov Report

Merging #25813 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25813      +/-   ##
==========================================
+ Coverage   91.27%   91.27%   +<.01%     
==========================================
  Files         173      173              
  Lines       53002    53002              
==========================================
+ Hits        48375    48378       +3     
+ Misses       4627     4624       -3

Flag	Coverage Δ
#multiple	`89.85% <100%> (+0.01%)`	⬆️
#single	`41.77% <0%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/computation/expr.py	`89.27% <100%> (+0.74%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fbe2523...64004bc. Read the comment docs.

codecov · 2019-03-21T02:14:05Z

Codecov Report

Merging #25813 into master will decrease coverage by 0.62%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25813      +/-   ##
==========================================
- Coverage   91.89%   91.27%   -0.63%     
==========================================
  Files         175      173       -2     
  Lines       52509    53002     +493     
==========================================
+ Hits        48255    48378     +123     
- Misses       4254     4624     +370

Flag	Coverage Δ
#multiple	`89.85% <100%> (-0.6%)`	⬇️
#single	`41.77% <0%> (+0.89%)`	⬆️

Impacted Files	Coverage Δ
pandas/core/computation/expr.py	`89.27% <100%> (-7.43%)`	⬇️
pandas/io/gbq.py	`25% <0%> (-62.5%)`	⬇️
pandas/compat/__init__.py	`58.03% <0%> (-19.47%)`	⬇️
pandas/io/common.py	`72.86% <0%> (-18.97%)`	⬇️
pandas/io/excel/_util.py	`78.82% <0%> (-8.68%)`	⬇️
pandas/compat/chainmap.py	`61.9% <0%> (-4.77%)`	⬇️
pandas/core/groupby/categorical.py	`95.45% <0%> (-4.55%)`	⬇️
pandas/core/dtypes/cast.py	`88.16% <0%> (-3.2%)`	⬇️
pandas/io/s3.py	`86.36% <0%> (-3.12%)`	⬇️
... and 115 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a15069...223e034. Read the comment docs.

WillAyd

Can you add a whatsnew note for v0.25?

…5813)

doc/source/whatsnew/v0.25.0.rst

pandas/tests/computation/test_eval.py

WillAyd · 2019-04-02T17:09:00Z

@dtpc can you merge master and address comments?

jreback · 2019-04-05T00:30:52Z

can you merge master and update

…25813)

WillAyd · 2019-04-17T16:13:57Z

doc/source/whatsnew/v0.25.0.rst

@@ -412,6 +412,7 @@ Other

 - Improved :class:`Timestamp` type checking in various datetime functions to prevent exceptions when using a subclassed `datetime` (:issue:`25851`)
 - Bug in :class:`Series` and :class:`DataFrame` repr where ``np.datetime64('NaT')`` and ``np.timedelta64('NaT')`` with ``dtype=object`` would be represented as ``NaN`` (:issue:`25445`)
+- Bug in :class:`BaseExprVisitor` which caused :func:`eval` expressions to fail when named keyword arguments were included within the expression string (:issue:`25813`)


Still needs a little work to make this user facing, i.e. only references items in the documented API (which BaseExprVisitor is not).

Would be better referencing the DataFrame.query method

WillAyd · 2019-04-17T16:15:48Z

pandas/tests/computation/test_eval.py

@@ -1757,6 +1757,29 @@ def test_no_new_globals(self, engine, parser):
        assert gbls == gbls2


+class TestEvalNamedKWargs(object):


I don't think we need a class here; OK to remove this and put tests at top level of module

WillAyd · 2019-04-17T16:16:46Z

pandas/tests/computation/test_eval.py

+        df = pd.DataFrame([["I", "XYZ"], ["IJ", None]], columns=['A', 'B'])
+
+        expected = df[df["A"].str.contains("J")]
+        result = df.query("A.str.contains('J')", engine="python",


Any reason to not parametrize this for all engines?

Yes, because engine=pandas fails for this test (though that should have been explicit).

It's tries to hash a Series object.

WillAyd · 2019-04-17T16:17:13Z

pandas/tests/computation/test_eval.py

@@ -1757,6 +1757,29 @@ def test_no_new_globals(self, engine, parser):
        assert gbls == gbls2


+class TestEvalNamedKWargs(object):
+    # xref https://github.com/pandas-dev/pandas/issues/25813


Just add #GH 25813 to the first line of the tests added here

jreback · 2019-05-12T21:19:00Z

@pandas-dev/pandas-core if someone wants to finish up

jreback · 2019-05-19T18:28:30Z

this PR is superseded by #26426

dtpc added 2 commits March 21, 2019 11:07

TST: Test df.query with "str.contains()"

8f22ebc

BUG: Fix bug when using eval to call "str" methods

64004bc

dtpc changed the title ~~Fix/eval str contains~~ BUG: Fix bug when using df.query with "str.contains()" Mar 21, 2019

WillAyd requested changes Mar 21, 2019

View reviewed changes

DOC: Update whatsnew v0.25.0 for named kwarg eval fix (pandas-dev#2…

087157c

…5813)

dtpc force-pushed the fix/eval_str_contains branch from 233a26b to 087157c Compare March 21, 2019 03:31

DOC: Remove trailing whitespace

36b0f43

WillAyd requested changes Mar 21, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Outdated Show resolved Hide resolved

jreback requested changes Mar 22, 2019

View reviewed changes

pandas/tests/computation/test_eval.py Outdated Show resolved Hide resolved

jreback added the Indexing Related to indexing on series/frames, not to indexes themselves label Mar 22, 2019

dtpc added 2 commits April 13, 2019 13:17

Merge remote-tracking branch 'pandas/master' into fix/eval_str_contains

63122dc

TST: Test explicitly pd.eval with named keyword argument (pandas-dev#…

223e034

…25813)

WillAyd requested changes Apr 17, 2019

View reviewed changes

jreback closed this May 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Fix bug when using df.query with "str.contains()" #25813

BUG: Fix bug when using df.query with "str.contains()" #25813

dtpc commented Mar 21, 2019 •

edited

Loading

codecov bot commented Mar 21, 2019

codecov bot commented Mar 21, 2019 •

edited

Loading

WillAyd left a comment

WillAyd commented Apr 2, 2019

jreback commented Apr 5, 2019

WillAyd Apr 17, 2019

WillAyd Apr 17, 2019

WillAyd Apr 17, 2019

gfyoung May 18, 2019 •

edited

Loading

WillAyd Apr 17, 2019

jreback commented May 12, 2019

jreback commented May 19, 2019

		@@ -1757,6 +1757,29 @@ def test_no_new_globals(self, engine, parser):
		assert gbls == gbls2


		class TestEvalNamedKWargs(object):

BUG: Fix bug when using df.query with "str.contains()" #25813

BUG: Fix bug when using df.query with "str.contains()" #25813

Conversation

dtpc commented Mar 21, 2019 • edited Loading

codecov bot commented Mar 21, 2019

Codecov Report

codecov bot commented Mar 21, 2019 • edited Loading

Codecov Report

WillAyd left a comment

Choose a reason for hiding this comment

WillAyd commented Apr 2, 2019

jreback commented Apr 5, 2019

WillAyd Apr 17, 2019

Choose a reason for hiding this comment

WillAyd Apr 17, 2019

Choose a reason for hiding this comment

WillAyd Apr 17, 2019

Choose a reason for hiding this comment

gfyoung May 18, 2019 • edited Loading

Choose a reason for hiding this comment

WillAyd Apr 17, 2019

Choose a reason for hiding this comment

jreback commented May 12, 2019

jreback commented May 19, 2019

dtpc commented Mar 21, 2019 •

edited

Loading

codecov bot commented Mar 21, 2019 •

edited

Loading

gfyoung May 18, 2019 •

edited

Loading