BUG: query uses numexpr even when 'compute.use_numexpr' is False

[quangngd]

• closes Dataframe.query uses numexpr even when 'compute.use_numexpr' is False #32556
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[jreback]

I am not sure how this address the actual issue (which is that numexpr knows nothing about extension types; they shouldn't be passed to it at all).

but the title suggests you think there is an actual but in use_numexpr? can you explain.

[quangngd]

If i understand correctly, the issue #32556 is about df.query does not care about the pd.set_option option, not about the extension dtype (which is still open at #25369)
@teto used pd.set_option('compute.use_numexpr', False) as a workaround for the original dtype issue but turned out it is also a bug on its own.

[teto]

@quangngd that's an accurate description of the issue. Thanks for taking care of this !

[jreback]

@quangngd you need to have a test which fails on master but works here. can you construct one.

[teto]

there is one in my issue (might not be usable out of the box though).

[quangngd]

@teto I don't think it's a good idea using a bug as a test as it would be fixed in the future.

[quangngd]

So to my understanding, the test must be able to differentiate the used engine during the df.query/ df.eval call (yah, just found out the bug is in df.eval too, and is too fixed by this pr). The thing is we could't use any "non-parse-able" behaviour since it could be a bug and could be fixed in the future.
The best I could come up with is the following based on numexpr' note.

import pandas as pd
df = pd.DataFrame(data=[4+1j,5,6], columns=["col_test"])
s = "abs(col_test)"
display(df.eval(s, engine='python').dtype)
display(df.eval(s, engine='numexpr').dtype)

output:

dtype('float64')
dtype('complex128')

But so far I can't think of how to make a test for df.query

[jreback]

@quangngd I am not sure of the status here. please merge master and address comments.

[quangngd]

@jreback the fix is already there but I cannot think of a sensible test

[rhshadrach]

@quangngd: What about this?

from pandas.core.computation.eval import _check_engine
with pd.option_context('compute.use_numexpr', False):
    result = _check_engine(None)
    assert result == 'python'

[quangngd]

@rhshadrach thankyou, the test indeed fails on master and pass on my branch. Where should be the proper place to place the test?

[rhshadrach]

Where should be the proper place to place the test?

tests.computation.test_eval

[jbrockmendel]

i think in this case we need to check expr.USE_NUMEXPR, not USE_NUMEXPR since the former may change at runtime

[jreback]

closing, this needs a reproducible tests that fails before any patch. (it can certainly be part of the patch, but demonstrate on the issue)

[teto]

wait what ? it's not easy to reproduce, you would need to disable one package or monkeypatch one package I suppose. The absence of a test should not prevent fixing an issue/document it/remove the feature.

[jreback]

w/o a test how do you know it was fixed?

[rhshadrach]

Is this test not sufficient?

#33006 (comment)

[jreback]

sure if it fails on master