ENH: fix eval scoping issues

[cpcloud]

closes #5987
closes #5087

Relevant user-facing changes:

In [9]: df = DataFrame(randn(5, 2), columns=list('ab'))

In [10]: a, b = 1, 2

In [11]: # column names take precedence

In [12]: df.query('a < b')
Out[12]:
          a         b
0 -0.805549 -0.090572
1 -1.782325 -1.594079
2 -0.984364  0.934457
3 -1.963798  1.122112

[4 rows x 2 columns]

In [13]: # we must use @ whenever we want a local variable

In [14]: df.query('@a < b')
Out[14]:
          a         b
3 -1.963798  1.122112

[1 rows x 2 columns]

In [15]: # we cannot use @ in eval calls

In [16]: pd.eval('@a + b')
  File "<string>", line unknown
SyntaxError: The '@' prefix is not allowed in top-level eval calls, please refer to your variables by name without the '@' prefix

In [17]: pd.eval('@a + b', parser='python')
  File "<string>", line unknown
SyntaxError: The '@' prefix is only supported by the pandas parser

• update query/eval docstrings/indexing/perfenhancing
• make sure docs build
• release notes
• pytables
• make the repr of Scope objects work or revert to previous version
• more tests for new local variable scoping API
• disallow (and provide a useful error message for) locals in expressions like pd.eval('@a + b')
• Raise when your variables have the same name as the builtin math functions that numexpr supports, since you cannot override them in numexpr.evaluate, even when explicitly passing them. For example

import numexpr as ne
sin = randn(10)
d = {'sin': sin}
result = ne.evaluate('sin > 1', local_dict=d, global_dict=d)
result == array(True)

For reference, after this PR local variables are given lower precedence than column names. For example

a, b = 1, 2
df = DataFrame(randn(10, 2), columns=list('ab'))
res = df.query('a > b')

will no longer raise an exception about overlapping variable names. If you want the local a (as opposed to the column a) you must do

df.query('@a > b')

[jreback]

oh boy! looks like the right way to do this..... 👍

[cpcloud]

Last issue is a scoping issue with py3 ... somehow not capturing scope from sys._getlevel.

[jreback]

guess this was not a small change 🔢

[cpcloud]

Nope. But scope is less of a hack and I think overall implementation is easier to understand. Actually got up early to try and fix this but all the back and forth of comparing py2 and 3 is huge time sink

[cpcloud]

turns out that list comprehensions work differently re the stack in python 3....each iteration generates a new stack frame. std for loops don't work that way so that should do it

[cpcloud]

Ok this is passing locally on py27 and py33

[cpcloud]

@jreback Whenever you get a chance .... this is passing.

[jreback]

won't pretend I understand! hahah

can you build the docs and just check that everything looks ok their?

I find sometimes they are good tests in and of themselves (aside which they need slight changing anyhow, right?)

[cpcloud]

Oh right! Those have to be changed anyway becuase of the new API ... let me add some new check boxes..

[cpcloud]

Biggest changes are

• Use of DeepChainMap (a nice wrapper around dictionaries that lets you store/search/update multiple dictionaries in a particular order). This is the most major change and most important since it obviates a lot of my scope hacks that were there before.
• local API is slightly different more explicit as we discussed
• stronger syntax checks for locals and a helpful err msg

[jreback]

maybe post on the top the relevant changes (from an ipython session).....

[cpcloud]

sure

[cpcloud]

@y-p @jreback @jorisvandenbossche Would be very grateful for any comments y'all might have.

[jreback]

your exceptions should be multi line (looking at the syntax error one for using a @ in eval)

[jreback]

u should put what u put in release notes in v0.14 API change section

[cpcloud]

done

[cpcloud]

adding a test for #5087. this closes that too!

[jreback]

u create temporaries for the locals to avoid name collisions?

[jreback]

looks fine to me
go ahead and squash

[cpcloud]

Temporaries are created for strings and lists

[cpcloud]

For locals I just rename them when the terms are created during parsing.

[jreback]

oh ok

looks fine 2 me otherwise

can merge when u r ready

[jorisvandenbossche]

Maybe you can leave out the ~ here (so just :func:pandas.eval``, this will show it in the docs as pandas.eval() instead of `eval()`) so it is clear it is not about the same eval as the paragraph above

[jorisvandenbossche]

maybe mention something of @ in the docstring?

[jorisvandenbossche]

I am not really familiar with this code, but just added some minor doc comments.

[cpcloud]

Thanks for the comments guys.

[cpcloud]

@jreback Should I squash some more? This is ready for merging if there are no more comments.

[jreback]

maybe a bit more?

[cpcloud]

Okay squashed and cleaned up the query docstring. Would like to keep some history so that I remember why I made change X especially some of these python version differences

[jreback]

cool....go for it

[cpcloud]

bombs away