String comparison in query()

[michaelbilow]

Hi, it seems that string comparisons aren't supported in query() yet, so maybe this isn't a bug yet. Anyway, hopefully this behavior will be fixed for future editions of pandas.

import pandas as pd
import numexpr as ne

a = list('abcdef')
b = range(6)
df = pd.DataFrame({'X':pd.Series(a),'Y': pd.Series(b)})

df_Y = df.query('Y < 3')             ## Works fine.
ne_works = ne.evaluate('"a" < "d"')  ## ne_works == np.array([True])
df_X = df.query('X < "d"')           ## RuntimeError: max recursion depth exceeded

[jreback]

cc @cpcloud

IIRC this is not implemented ATM

[cpcloud]

Yep only == and != are tested for strings.

[cpcloud]

Let me take a look at why this happens and I'll clean up the error message or I'll implement it.

[cpcloud]

Interesting ... it only fails on the numexpr engine ... yay for writing tests first!

[cpcloud]

Ok this was simply a case of making those evaluate in Python space when object dtypes are being compared....

I think maybe a doc note about the fact that expressions that are evaluated on a case by case basis in terms of the types of their result is needed ... I could've sworn I wrote something about this, but maybe just in our conversations ....

[jreback]

hahah...

[cpcloud]

@chuyelchulo Couple of details that might interest you:

• Expressions that would result in an object dtype (including simple variable evaluation) have to be evaluated in Python space. The main reason is for back compat with older versions of numpy that will truncate a string if you call astype(str) on them if they have elements with more than 60 chars. We can't pass object arrays to numexpr thus string comparisons are evaluated in Python space.
• The upshot is that that only applies to strings. So, if you have an expression for example, that's a string comparison and-ed together with another boolean expression that's from a numeric comparison, the numeric comparison will be evaluated by numpexpr. In fact, in general, query/eval will "pick out" the subexpressions that are eval-able by numexpr and those that must be evaluated in Python space transparently to the user.

[cpcloud]

I'm adapting that for the docs, PR coming shortly

[cpcloud]

@chuyelchulo Thanks for reporting this.

[michaelbilow]

Thanks, that was really interesting!