ENH: remove restrictions to numexpr to allow where etc.

[jonas-eschle]

Is your feature request related to a problem?

the evaluation of a query is currently limited to the list _mathops while numexpr would support more, most notably a where (that would also solve other issues simple).

I do not see any reason for this restriction. In fact, simply adding the where runs (at least for my use case). Why is this restriction in place? Why can't we enlarge it/directly pass it through to numexpr?

Describe the solution you'd like

Allow the full operator set that numexpr supports in the pd.eval

API breaking implications

Nothing

Alternatives

Using .where is an option if you can access the dataframe directly (although suboptimal). However, if your selection of data is based on passing a selection string around instead of the df (several reasons for this), the latter is not feasible.

The following doesn't work:

import pandas as pd

data = {'a': [1, 2, 3]}                                                 

df = pd.DataFrame({'a': [1, 2, 3]})
df.eval('where(a>2, 42, 0)')

whereas in numexpr it does

numexpr.evaluate('where(a>2, 42, 0)', local_dict=data)

we expect this to return [0, 0, 42]

[TomAugspurger]

Can you add a minimal example? http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[jonas-eschle]

Well, yeah. I've updated it. But it does not make so much sense I think to provide an example for eval/query? Or what exactly do you think is unclear?

Again, in short: eval/query use numexpr to forward a string for evaluation but (to my view) unnecessarily limit the allowed expressions to a subset of what numexpr supports.

Let me know if things are confusing and thanks a lot for taking a look at it!

[TomAugspurger]

Thanks. And can you add the expected output?

Do you know if we have other expressions that would be supported by only one engine (numexpr in this case)?

[Liam3851]

I was about to report this issue but realized [edit: I would have created] a dupe. numexpr supports the following functions missing from the definitions in pandas.core.computation.ops:

• where
• tan (oddly, we support sin, cos, tanh, arctan, and arctan2, but not tan)
• conj, real, imag, complex (though I'm not sure the extent to which pandas supports complex numbers in general so this might be fine)
• contains (for strings)

Expectation is that

import pandas as pd

data = {'a': [1, 2, 3]}                                                 

df = pd.DataFrame({'a': [1, 2, 3]})
df.eval('where(a>2, 42, 0)')

should return the same as

pd.DataFrame({'a':[0, 0, 42]})

[Liam3851]

My apologies @mayou36 can you reopen? I meant that I would have created a dupe. This is not a dupe, I think it's a legit issue that pandas doesn't support where or tan when numexpr does.

[jonas-eschle]

Oc, sorry that was by mistake and it is not yet resolved.

@TomAugspurger, do you have an idea why this is not here?

[achimgaedke]

I have "solved" my problem by using

import pandas
pandas.core.computation.ops.MATHOPS = (*pandas.core.computation.ops.MATHOPS, "where")

Works out of the box with ternary operators, 🎉

df = pandas.DataFrame({"a": [2.0, 4.0, 5.0]})
pandas.eval("where(df.a > 3.0, df.a, 1)", target=df)

results in

0    1.0
1    4.0
2    5.0
Name: done, dtype: float64

Not exactly proud of this solution, but this shows that this feature request is probably done by adding the strings and writing some unit tests.

NB: arguments 2 and 3 have to be numbers - I'd love to have strings (same type should only be required only for arg 2 and 3)
NB2: (df.a>3.0) * df.a + (df.a<=3) * 1 works like where(df.a > 3.0, df.a, 1)

[jonas-eschle]

@TomAugspurger any news on this? This seems to be a limitation for no apparent reason?

[TomAugspurger]

I’m not sure, but it looks like the issue is still open.

…

On Apr 5, 2024, at 10:10 AM, Jonas Eschle ***@***.***> wrote: @TomAugspurger <https://github.com/TomAugspurger> any news on this? This seems to be a limitation for no apparent reason? — Reply to this email directly, view it on GitHub <#34834 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIXLXMFWMZ3DEHEWNA3Y325FVBFKMF2HI4TJMJ2XIZLTS2BKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFUU3TMOBRGKSG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLKJSGE4TSNRQG42TRJDOMFWWLKLIMFZV63DBMJSWZAVFOZQWY5LFVIYTMMZRHE4DKMRWG6SG4YLNMWUWQYLTL5WGCYTFNSWHG5LCNJSWG5C7OR4XAZNMJFZXG5LFINXW23LFNZ2KM5DPOBUWG44VQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLJRYGU4DCMRXQKSHI6LQMWSWS43TOVS2K5TBNR2WLKJWGQYDAMZRGE4TLAVEOR4XAZNFNRQWEZLMUV3GC3DVMWSTONRYGEZIFJDUPFYGLJLMMFRGK3FFOZQWY5LFVEZDCOJZGYYDONJYQKSHI6LQMWSWYYLCMVWKK5TBNR2WLKRRGYZTCOJYGUZDMN5HORZGSZ3HMVZKMY3SMVQXIZI>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jonas-eschle]

Okay, so what about if I go ahead and remove the limitation in a PR? Maybe the tests will tell us why this is there, or some maintainer, but without comments in the code or on the issue here, we can only guess.

[Aloqeely]

Another issue was opened for this same topic: #55091

@jonas-eschle I suggest you add whichever functions you think are going to be useful and write unit tests that ensures every function you're adding is working properly.
A PR was submitted recently to add tan (#58334), so probably wait until it gets reviewed/merged

[domsmrz]

FYI also loosely related I've filed #58329 which means that df.eval('where(...)') returns np.array for engine="python" and pd.Series for engine="numexpr" (if one adds "where" to the list of allowed functions). Probably not necessarily a blocker, but something to keep in mind (at least while writing tests).