Numexpr 2.4.6 #15383

FrancescAlted · 2017-02-13T10:52:43Z

closes DEPS: drop numexpr < 2.4.6 #15213
passes git diff upstream/master | flake8 --diff
whatsnew entry

This addresses #15213. Actually, the recommended version for me is 2.6.2, but I understand that this is too recent still.

jorisvandenbossche · 2017-02-13T11:21:54Z

@FrancescAlted we have a few occurences of numpexpr pinned at older versions in ci/requirement files (I think the 2.7_COMPAT.run and 3.4_SLOW.run ones). Best to change those versions to 2.4.6 then.

jreback · 2017-02-13T11:39:06Z

doc/source/install.rst

@@ -226,7 +226,7 @@ Recommended Dependencies

 * `numexpr <https://github.com/pydata/numexpr>`__: for accelerating certain numerical operations.
  ``numexpr`` uses multiple cores as well as smart chunking and caching to achieve large speedups.
-  If installed, must be Version 2.1 or higher (excluding a buggy 2.4.4). Version 2.4.6 or higher is highly recommended.
+  If installed, must be Version 2.4.6 or higher.  Numexpr 2.6.2 or higher is strongly recommended.


you don't need the recommended part

Well, IMO the improvements in https://github.com/pydata/numexpr/blob/master/RELEASE_NOTES.rst#changes-from-261-to-262 are worth a recommendation, but I can remove it if you insist.

it may be but back compat is more important

jreback · 2017-02-13T11:39:58Z

doc/source/whatsnew/v0.20.0.txt

 --------------------

 This is a major release from 0.19 and includes a small number of API changes, several new features,
 enhancements, and performance improvements along with a large number of bug fixes. We recommend that all
 users upgrade to this version.

+.. warning::
+
+   Due to important fixes and performance improvements, ``numexpr`` version is now required to be >= 2.4.6 and it will not be used at all if this requisite is not fullfilled (:issue:`15213`).


i would just move this to the other api changes section; just say 2.4.6 is now required

You probably mean "Other enhancements" section (numexpr does not introduce any API change).

it's an API change for pandas meaning if u has a version that is no longer supported then things will change - its also something to be aware of when upgrading

jreback · 2017-02-13T11:40:42Z

pandas/computation/__init__.py

-    # has some hard-to-diagnose bugs
-    if ver == LooseVersion('2.4.4'):
-        _NUMEXPR_INSTALLED = False
-        warnings.warn(


there is a test for this i think

Yep. Addressed in f225598.

and fixed a typo in 7a275ce

codecov-io · 2017-02-13T11:43:44Z

Codecov Report

Merging #15383 into master will decrease coverage by -0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #15383      +/-   ##
==========================================
- Coverage   90.42%   90.41%   -0.02%     
==========================================
  Files         134      134              
  Lines       49357    49375      +18     
==========================================
+ Hits        44632    44640       +8     
- Misses       4725     4735      +10

Impacted Files	Coverage Δ
pandas/computation/init.py	`90.9% <100%> (+13.98%)`	✅
pandas/io/gbq.py	`16.46% <ø> (-0.42%)`	❌
pandas/core/generic.py	`96.33% <ø> (ø)`	✅
pandas/core/internals.py	`94.15% <ø> (ø)`	✅
pandas/core/missing.py	`84.95% <ø> (+0.19%)`	✅

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d9e75c7...c417fe2. Read the comment docs.

FrancescAlted · 2017-02-13T19:24:49Z

There are a couple of "allowed failures" in TravisCI. Any hint on what's going on there?

jorisvandenbossche · 2017-02-13T22:55:45Z

For the MacOSX one, I see

UnsatisfiableError: The following specifications were found to be in conflict:
  - numexpr 2.4.6
  - pytables 3.0.0 -> numexpr 2.2*

FrancescAlted · 2017-02-14T09:13:18Z

Uh, I see the Mac OSX build that completed just fine. Also, the AppVeyor failure seems like a transitory failure during conda packages access:

conda.exceptions.CondaRuntimeError: Runtime error: RuntimeError: Runtime error: HTTPError: 502 Server Error: Bad Gateway for url: https://repo.continuum.io/pkgs/free/win-64/xlsxwriter-0.9.6-py35_0.tar.bz2: https://repo.continuum.io/pkgs/free/win-64/xlsxwriter-0.9.6-py35_0.tar.bz2

But I am still unsure on the level of importance of the "allowed failures" in TravisCI.

jorisvandenbossche · 2017-02-14T09:28:16Z

Sorry, I mislooked, not the MacOSX build of course, build number 6 with the old versions (2.7_COMPAT), the error now is:

UnsatisfiableError: The following specifications were found to be in conflict:
  - numexpr 2.4.6 -> numpy 1.10*
  - numpy 1.7.1
Use "conda info <package>" to see the dependencies for each package.

We want to keep the numpy 1.7 there for now (although there is an issue to bump that requirement as well), so I would just keep the older numexpr in this build (then it is a test that you get the correct error for that).

But I am still unsure on the level of importance of the "allowed failures" in TravisCI.

They are not 'allowed' to fail in principle (only for specific things, eg compat issue with numpy master that is not yet fixed). I think it is like that to get quicker feedback from travis.

There is still a lint error:

pandas/computation/__init__.py:18:17: E128 continuation line under-indented for visual indent

The appveyor issue indeed does not seem to be related to this PR.

…rements

jorisvandenbossche · 2017-02-14T10:18:22Z

@FrancescAlted Was it needed to also revert the version change in the 3.4_SLOW build? As this one was not failing I think.

FrancescAlted · 2017-02-14T10:35:56Z

Yes, it was: https://travis-ci.org/pandas-dev/pandas/builds/201238220. I am not familiar with the convention to run different configurations, but I'd say that stems from: https://github.com/pandas-dev/pandas/blob/master/.travis.yml#L131. I just reverted the numexpr versions existing in this two scenarios. My intention is to look at the possible errors after the revert and try to fix them.

jreback · 2017-02-14T13:28:02Z

i would change this version to 2.4.6, otherwise lgtm. https://github.com/pandas-dev/pandas/blob/master/ci/requirements-3.4_SLOW.run

ping on green.

FrancescAlted · 2017-02-14T15:29:47Z

I think we will still have a remaining error here. But the assert will not raise the error as such, and we would need something like from pandas.computation import __NUMEXPR_INSTALLED. However, there is a line at the beginning saying from pandas.computation import _MIN_NUMEXPR_VERSION which already raises the warning, and hence the error is not re-raised. Not sure how to cope with that. Opinions?

jorisvandenbossche · 2017-02-14T15:47:22Z

So probably that test never worked .. What you could do is move the import inside the test and in a "catch warnings" context, and then check that a warning is raised in the specific case and not otherwise (https://docs.python.org/3.5/library/warnings.html#testing-warnings)?

jreback · 2017-02-14T16:33:20Z

no it worked but the import process is changed by this PR
it needs to be protected

FrancescAlted · 2017-02-14T19:38:28Z

I have been looking at the original code, and I don't think it worked because assert not _NUMEXPR_INSTALLED cannot possibly raise a Warning; can it?

jreback · 2017-02-14T21:11:05Z

so I think with the changed up order of tests, this test_compat that is failing (on 2.7_COMPAT) is not able to assert the warning as its already been produced (you can see in the output actually).

So I guess just change the test then.

FrancescAlted · 2017-02-15T09:07:45Z

Ok, I think that I have found the issue and fixed it in e1b34a9. Basically, one needs to force the reload() of pandas.computation module to trigger the UserWarning in import time. Hopefully we should see all green after this one.

FrancescAlted · 2017-02-15T10:54:10Z

Well, it turns out that reload() is tricky and it still cannot raise the UserWarning in Python 2.7 (although 3.4 seems to work well). After googling a bit, I have read that reload() is very tricky to get right, so my vote is to remove this UserWarning test completely (unless you have a better proposition).

jreback · 2017-02-15T13:14:33Z

@FrancescAlted I would just remove the check for the warning (leave the test just don't assert that it produces)

jreback · 2017-02-15T13:15:00Z

pandas/tests/computation/test_compat.py

@@ -10,6 +10,12 @@

 from pandas.computation.engines import _engines
 import pandas.computation.expr as expr
+from pandas.computation import _MIN_NUMEXPR_VERSION
+# Get reload for Python 3.4 and on, if not, use internal reload()


too complicated, don't reload

jreback · 2017-02-15T13:15:16Z

pandas/tests/computation/test_compat.py

-        if ver == LooseVersion('2.4.4'):
-            assert not _NUMEXPR_INSTALLED
-        elif ver < LooseVersion('2.1'):
+        if ver < LooseVersion(_MIN_NUMEXPR_VERSION):
            with tm.assert_produces_warning(UserWarning,


just remove this assert_produces_warning

jreback · 2017-02-15T13:15:22Z

pandas/tests/computation/test_compat.py

            with tm.assert_produces_warning(UserWarning,
                                            check_stacklevel=False):
-                assert not _NUMEXPR_INSTALLED
+                reload(pd.computation)


too complicated

jreback · 2017-02-15T13:34:49Z

lgtm. ping on green.

FrancescAlted · 2017-02-15T13:35:12Z

Done. Hopefully this will give us all green (bar some temporary failures that apparently are usual in TravisCI).

jreback · 2017-02-15T13:38:58Z

@FrancescAlted well there have been some flakiness with some gbq tests; I turned them off for now. We should have all green.

FYI we just have the fail fast to make the results appear quicker. We have LOTS of configurations to test. So allowed failures are really NOT allowed :>

jreback · 2017-02-15T13:40:21Z

one more FYI. every push on a PR generates a new travis run (which can take hours). So generally like to bunch them up if possible. This PR of course is a case that its necessary to test little changes so no big deal.

jreback · 2017-02-15T15:21:20Z

thanks @FrancescAlted

closes pandas-dev#15213 Author: Francesc Alted <[email protected]> Closes pandas-dev#15383 from FrancescAlted/numexpr-2.4.6 and squashes the following commits: c417fe2 [Francesc Alted] Simplify and remove UserWarning testing on numexpr import e1b34a9 [Francesc Alted] Force a reload of pd.computation for actually triggering the UserWarning c081199 [Francesc Alted] Relax the exact message for the ImportError 73f0319 [Francesc Alted] numexpr requisite raised to 2.4.6 0d4ab9a [Francesc Alted] Restored the old numexpr version dependencies to adjust for old requirements c1aae19 [Francesc Alted] Fixed a lint error 7575ba2 [Francesc Alted] Using constants instead of literals for numexpr version 7a275ce [Francesc Alted] Fixed a typo 93f54aa [Francesc Alted] numexpr section moved to Other API changes section 3b6e58b [Francesc Alted] Removed recomendation for numexpr 2.6.2 f225598 [Francesc Alted] Updated test_compat for numexpr 2.4.6 8bd4ed1 [Francesc Alted] numexpr 2.4.6 requirement moved to other enhancements section e45b742 [Francesc Alted] Moved pinned versions in CI folder to 2.4.6 6e12e29 [Francesc Alted] Added a notice on the recommended numexpr version ac62653 [Francesc Alted] Require numexpr 2.4.6 ab79c54 [Francesc Alted] Require numexpr 2.6.2

Francesc Alted added 3 commits February 13, 2017 10:36

Require numexpr 2.6.2

ab79c54

Require numexpr 2.4.6

ac62653

Added a notice on the recommended numexpr version

6e12e29

FrancescAlted mentioned this pull request Feb 13, 2017

Require numexpr 2.6.2 #15382

Closed

3 tasks

jorisvandenbossche modified the milestones: No action, 0.20.0 Feb 13, 2017

jorisvandenbossche added the Dependencies Required and optional dependencies label Feb 13, 2017

jreback requested changes Feb 13, 2017

View reviewed changes

Moved pinned versions in CI folder to 2.4.6

e45b742

Francesc Alted added 6 commits February 13, 2017 13:00

numexpr 2.4.6 requirement moved to other enhancements section

8bd4ed1

Updated test_compat for numexpr 2.4.6

f225598

Removed recomendation for numexpr 2.6.2

3b6e58b

numexpr section moved to Other API changes section

93f54aa

Fixed a typo

7a275ce

Using constants instead of literals for numexpr version

7575ba2

Francesc Alted added 2 commits February 14, 2017 10:35

Fixed a lint error

c1aae19

Restored the old numexpr version dependencies to adjust for old requi…

0d4ab9a

…rements

jreback approved these changes Feb 14, 2017

View reviewed changes

Francesc Alted added 2 commits February 14, 2017 15:43

numexpr requisite raised to 2.4.6

73f0319

Relax the exact message for the ImportError

c081199

Force a reload of pd.computation for actually triggering the UserWarning

e1b34a9

jreback reviewed Feb 15, 2017

View reviewed changes

Simplify and remove UserWarning testing on numexpr import

c417fe2

jreback closed this in 2372d27 Feb 15, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Numexpr 2.4.6 #15383

Numexpr 2.4.6 #15383

FrancescAlted commented Feb 13, 2017 •

edited by jreback

Loading

jorisvandenbossche commented Feb 13, 2017

jreback Feb 13, 2017

FrancescAlted Feb 13, 2017

jreback Feb 13, 2017

jreback Feb 13, 2017

FrancescAlted Feb 13, 2017

jreback Feb 13, 2017

jreback Feb 13, 2017

FrancescAlted Feb 13, 2017

FrancescAlted Feb 13, 2017

codecov-io commented Feb 13, 2017 •

edited

Loading

FrancescAlted commented Feb 13, 2017

jorisvandenbossche commented Feb 13, 2017

FrancescAlted commented Feb 14, 2017

jorisvandenbossche commented Feb 14, 2017

jorisvandenbossche commented Feb 14, 2017

FrancescAlted commented Feb 14, 2017

jreback commented Feb 14, 2017

FrancescAlted commented Feb 14, 2017

jorisvandenbossche commented Feb 14, 2017

jreback commented Feb 14, 2017

FrancescAlted commented Feb 14, 2017

jreback commented Feb 14, 2017

FrancescAlted commented Feb 15, 2017

FrancescAlted commented Feb 15, 2017

jreback commented Feb 15, 2017

jreback Feb 15, 2017

jreback Feb 15, 2017

jreback Feb 15, 2017

jreback commented Feb 15, 2017

FrancescAlted commented Feb 15, 2017

jreback commented Feb 15, 2017

jreback commented Feb 15, 2017

jreback commented Feb 15, 2017

Numexpr 2.4.6 #15383

Numexpr 2.4.6 #15383

Conversation

FrancescAlted commented Feb 13, 2017 • edited by jreback Loading

jorisvandenbossche commented Feb 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Feb 13, 2017 • edited Loading

Codecov Report

FrancescAlted commented Feb 13, 2017

jorisvandenbossche commented Feb 13, 2017

FrancescAlted commented Feb 14, 2017

jorisvandenbossche commented Feb 14, 2017

jorisvandenbossche commented Feb 14, 2017

FrancescAlted commented Feb 14, 2017

jreback commented Feb 14, 2017

FrancescAlted commented Feb 14, 2017

jorisvandenbossche commented Feb 14, 2017

jreback commented Feb 14, 2017

FrancescAlted commented Feb 14, 2017

jreback commented Feb 14, 2017

FrancescAlted commented Feb 15, 2017

FrancescAlted commented Feb 15, 2017

jreback commented Feb 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 15, 2017

FrancescAlted commented Feb 15, 2017

jreback commented Feb 15, 2017

jreback commented Feb 15, 2017

jreback commented Feb 15, 2017

FrancescAlted commented Feb 13, 2017 •

edited by jreback

Loading

codecov-io commented Feb 13, 2017 •

edited

Loading