Extension dtypes do not work with dataframe query method

[techvslife]

In the latest pandas version (0.24.1), extension dtypes do not work with numexpr (e.g. the “query” method on dataframes fails when the engine parameter is set to the default ):

https://stackoverflow.com/questions/54759936/extension-dtypes-in-pandas-appear-to-have-a-bug-with-query

Code to reproduce:
df_test = pd.DataFrame(data=[4,5,6], columns=["col_test"])
df_test = df_test.astype(dtype={"col_test": pd.Int32Dtype()})
df_test.query("col_test != 6")
Last lines of the long error message are:

File "...\site_packages\numexpr\necompiler.py", line 822, in evaluate zip(names, arguments)] File "...\site_packages\numexpr\necompiler.py", line 821, in signature = [(name, getType(arg)) for (name, arg) in File "...\site_packages\numexpr\necompiler.py", line 703, in getType raise ValueError("unknown type %s" % a.dtype.name) ValueError: unknown type object

[jreback]

you would have to explicitly add support; this is not a bug rather an enhancement requests

[techvslife]

Thank you. It may be that there should be a warning about this in the docs (until support is added in numexpr).

[techvslife]

fyi at numexpr, was mentioned that pandas may want to raise an error on unrecognized or unsupported dtypes in the query method rather than let it travel through. or could auto-fallback to pandas engine so as not to cause errors at all. but I assume there would be a downside to doing that check in adding overhead. —so not necessarily recommending but only mentioning it for consideration. Thanks.

[gfyoung]

but I assume there would be a downside to doing that check in adding overhead. —so not necessarily recommending but only mentioning it for consideration

Better error messages wouldn't hurt I think. Feel free to try implementing that.

[teto]

very annoying. Is there a way to setup the eval engine globally? set 'python' as default instead of numexpr ? Right now I have to build environments without numexpr to force panda to fall back on 'python' :s

[teto]

For a moment I thought I had found the graal with pd.set_option('compute.use_numexpr', False) but this doesn't disable numexpr in query/eval it seems so I still have to change my code to add engine="python" ?

[teto]

I checked if this was fixed with pandas 1.0 but it isn't. What I am still puzzled by and wonder if it warrants a new issue is that I disable numexpr via pd.set_option('compute.use_numexpr', False), my check log.debug("use numexpr? %d" % pd.get_option('compute.use_numexpr',)) confirms it is false yet pandas insists on using the numexpr engine. Do I use the API wrong or is it a bug ?

[mroeschke]

xref #29618