COMPAT: Consider dropping numpy 1.6 support

[cpcloud]

NumPy 1.6

NumPy 1.6 introduced numerous bugs around datetimes, forcing a plethora of hacks, and causing numerous stress-induced ulcers for an uncountably infinite number of people.

The time for pandas to lose this vestigial organ has come. The main reason for this is that datetime support is just broken, in nearly every sense of the word. Take a look at some of the conversion code in pandas/core/common.py and you'll see what I'm talking about.

I think we've probably reached a point where supporting numpy 1.6 is just going to cause more pain than it's worth. To me it makes no sense to want to keep upgrading pandas while keeping numpy <= 1.6.1. If you want modern pandas, you need to use modern-ish versions of its dependencies.

[jreback]

i think i raised this for #5108, but maybe its time

[jreback]

cc @yarikoptic

can you give an indication when major distros moving to 1.7 for their stable? (iirc somewhere jessie is doing this in oct of this year?)

[jtratner]

To me it makes no sense to want to keep upgrading pandas while keeping numpy <= 1.6.1. If you want modern pandas, you need to use modern-ish versions of its dependencies.

Excellent point. 👍

[yarikoptic]

here is the current state of numpy:

Ubuntu   1:1.3.0-3build1    http://packages.ubuntu.com/lucid/python-numpy
Ubuntu   1:1.6.1-6ubuntu1   http://packages.ubuntu.com/precise/python-numpy
Ubuntu   1:1.7.1-1ubuntu1   http://packages.ubuntu.com/raring/python-numpy
Ubuntu   1:1.7.1-1ubuntu1   http://packages.ubuntu.com/saucy/python-numpy
Ubuntu   1:1.8.1-1ubuntu1   http://packages.ubuntu.com/trusty/python-numpy
Ubuntu   1:1.8.1-1ubuntu1   http://packages.ubuntu.com/utopic/python-numpy
Debian   1:1.4.1-5          http://packages.debian.org/squeeze/python-numpy
Debian   1:1.6.2-1.2        http://packages.debian.org/wheezy/python-numpy
Debian   1:1.8.1-1+b1       http://packages.debian.org/jessie/python-numpy
Debian   1:1.8.1-1+b2       http://packages.debian.org/sid/python-numpy

so jessie will have the "modern" numpy, but current stable is at 1.6.2 and current pandas still happy on it. It would be great if dropping it could may be postponed after the next pandas release? according to reasoning above current release has broken handling of datetime on wheezy with numpy 1.6.2 -- then "dropping" support of numpy 1.6 in the next release would simply leave people on stable debian with a broken pandas, or have I misread it?

[jreback]

@yarikoptic you have misread.

pandas 'fixes' the timedelta support on < 1.7 numpy (using pandas operations). even though numpy is hopelessly broken (its much better in 1.8/1.9). In reality we now DO raise on < 1.7 numpy for unsupported ops.

the issue is that we have lots of test that are skipped on numpy < 1.7 and a fair number of ugly work-arounds for these.

I actually don't see dropping support for < 1.7 as much of a problem, because this is ONLY an issue if:

• you have a fairly old build (i use wheezy so am guilty!)
• you install/upgrade to current pandas using PIP
• you don't install numpy but instead use the system installed version

why ANYONE would do this is beyond me. If you are going to install pandas latest, what's the big deal with install numpy current?

[jreback]

@cpcloud let's separate the 2.6 part of this to a new issue

[dsm054]

Now that Python has officially blessed virtual environments with the adoption of venv, the canonical response to "but my system only has library version X installed" is now "so?"

+1 in favour of dropping broken (for our use cases) numpy.

[jreback]

@cpcloud can you put this on the mailing list as well, just to see if we are missing anything here. (and copy numpy-discussion too).

[cpcloud]

Sure

[yarikoptic]

@jreback regarding three bullet points. "you install/upgrade to current pandas using PIP" -- I agree that if someone does it, he is on his own ;) I was primarily concerned to have a reasonably good version within NeuroDebian for wheezy for at least another year (so people have time to upgrade from wheezy to jessie). If you think that 0.14.0+git17-g3849d5d-1~nd70+1 should be fine -- then go ahead with dropping the support. If you feel that it might be worth making a quick point or snapshot release before you drop 1.6.2 numpy support -- might even be better ;-)

[jreback]

@yarikoptic we are planning on releasing 0.14.1 tomorrow, which includes < 1.7 support. @cpcloud mentioned this for 0.15.0 approx due end of sept 2014. That work?

[yarikoptic]

Sounds great to me! Thank you guys!

[yarikoptic]

BTW FWIW and FYI -- if 0.15.0 comes out in Aug and would be polished enough -- it might still make it into jessie which is to start freezing in Sep . Otherwise jessie would come out with 0.14.1.

[cpcloud]

Mailing list seems to be a bit silent on this one. FWIW conda ships with numpy 1.8.1 and It seemed like a lot of people at SciPy 2014 were using conda. That said, I'm sure there are stragglers out there. I still don't understand why anyone would keep their numpy at <= 1.6.x and continue to follow pandas releases.

[shoyer]

I agree, I don't see a need to keep around support for ancient versions of numpy at this point.

[jreback]

ok, let's do this one. I think its time.

@cpcloud ?

prob need to search the code and remove stuff that is not necessary (and a lot of skipping tests)

so change builds 2.6/2.7_locale to numpy 1.7.0 I guess? (I have to create wheels, but no biggie)

[cpcloud]

alrighty, into the rabbit hole i go

[jreback]

their is actually some support for < 1.6 too (grin _np_version_under1p6)....should go too!

[jreback]

also need to fix setup too

[cpcloud]

yep there's quite a bit of acrobatics around dates as u know! i'll gut as mcuh as i can and then we can slowly remove things over time if need be

[cpcloud]

i put 1.7.0 on 2.6 and 1.7.1 on 2.7_locale

[jreback]

acrobatics in tseries/timedeltas.py (and tslib.pyx) is where a lot of this is

[cpcloud]

yep

[jreback]

ok, i'll make wheels tonight

[cpcloud]

sounds good

[cpcloud]

actually the np under 1.7 constants can go too, since we're requiring that that be the case now

[jreback]

yep, you can prob just take them out, and run the tests and see where it breaks :)

[jreback]

maybe a nice warning in __init__.py as well (in case someone installs pandas somehow w/o pip and relies on a system numpy < 1.7)

[cpcloud]

good call

[cpcloud]

i think a raise would be better than warning even though most things would probably work, that's not a very convincing reason.... prevent foot shooting and all

[jreback]

awesome!!!

[cpcloud]

:) thanks whoohoo! no more np16!

[jreback]

hmm I think that I need to take it out of my windows test builds - yeh!