BLD: are correct versions of numpy being installed / statsmodel install (wheel issue?)

[jreback]

numpy 1.6.1 supposed to be installed here ?
https://travis-ci.org/pydata/pandas/jobs/8782276

statsmodels not installing
https://travis-ci.org/pydata/pandas/jobs/8782274
#4128

[cpcloud]

looks like SITE_PKG_DIR is not set

[cpcloud]

it's trying to create a dir pandas in the root directory

[cpcloud]

k, put up a pr to add it back in

probably the version installed under full deps won't let pip install any other version

with np 1.6.1: https://travis-ci.org/cpcloud/pandas/jobs/8783101

let's see what happens with full deps...

[cpcloud]

well that was stupid...i forgot that apt removes deps too

[cpcloud]

lxml is also not being installed...

[cpcloud]

actually it is, in fact, a scikits.timeseries installation issue. it doesn't see that numpy is installed and installs the np1.7 wheel

[jreback]

ok then only have scikit timeseries if we have numpy 1.7?

[cpcloud]

numpy 1.6.1 gets installed

but then for some reason scikits overrides it even though pip recognizes that 1.6.1 is installed and even acknowledges that it knows that!

[cpcloud]

1.7 is only installed on python 3.3...so timeseries would only be run on python 3...but not really because timeseries is only for full deps where python 3.3 is not

[cpcloud]

why can't we just use the install that comes on the box? it uses 1.6.1

[cpcloud]

not sure about the version for py3 tho

[jreback]

I think the problem with the box install is that it compiles it....takes a while....

you could make the 3.3 full deps (much faster now anyhow)....and put scikit-timeseries only there....(or maybe even drop it......)

[cpcloud]

i meant the one installed by apt

[cpcloud]

but can't do that bc 1.6.1 is installed for py3.3 so that won't work

[jreback]

IIRC all the builds are np 1.7 except 3.3?

[cpcloud]

the other way around

[cpcloud]

all are 1.6.1 except 3.3 which is 1.7.0b2 at minimum

[jreback]

oh...i would make one of the 2.7 ones use 1.7.1 then its the stable production one

[cpcloud]

ok that works...then can use for scikits as well

[cpcloud]

arg patsy also requires numpy... -__-

[cpcloud]

i would rather not make all these changes and just figure out why pip is clobbering the previously installed versions...

[cpcloud]

gosh i'm slow. @y-p do we need the -I flag in PIP_ARGS?

[ghost]

try it. It was there to fix an issue, but that may have been due to a different problem.

[ghost]

Continuing from #4143 (comment).
travis does provide latest numpy preinstalled for all python versions: travis-ci/travis-cookbooks#48

[ghost]

re seperate versions based on underlying numpy, oh absolutely. Huge oversight on my part.
That's not possible to do by package naming only. There would need to be a package repo
per build configuration, and the proper url used.

How about you put together a requirements.txt with exact versions for each travis job, and
I'll setup the appropriate packages on the sever. would that work?

[cpcloud]

do u mean 5 separate requirements.txt files (one for each travis job)? or just one? not sure how one would work unless it's going to be parsed...

if each build has its own req file then you could do something like pip install -r requirements-$TRAVIS_PYTHON_VERSION-${TRAVIS_JOB_ID#*.}.txt and u might be able to knockout a huge chunk of ci/install.sh

[ghost]

yup, that was the idea. since each job has it's own version of dependencies,
it's better to manage it in a single config rather then a tangle of if clauses in a shell
script. and then each can have it's own package directory on the server, synched
with the requirements.txt. easier to automate as well.

[cpcloud]

quick before travis notices!

[cpcloud]

cpcloud@21740a4

[cpcloud]

oh hold on i used the wrong env var

[cpcloud]

cpcloud@8c64366

[cpcloud]

haha ur alter ego has awoken

[jreback]

stats models? not on the requirements list?

[cpcloud]

still in ci/install.sh. requires special handling bc of its dependence on pandas

[cpcloud]

vagrant up in speedpack?

[cpcloud]

not working...spits out a backtrace .... strange all my other vagrant boxes work fine

[cpcloud]

anyways i'll take it, but i can't really do it until tmrw

[cpcloud]

well...a few hours that is

[ghost]

please use @y-p.

it's building everything now, I'll upload when it's done,
I'd be glad if you would rework install.sh to point the script
at the correct urls.

vagrant 1.2.2.

[cpcloud]

@y-p looks like you deleted your comment above. i can't remember if there were any important details there...anyway it's up and working on my machine, is the idea that i can use it to test the wheels?

also, what else as far as URLs go needs to be done for install.sh? looks like travis is installing things correctly from wheels except this build: https://travis-ci.org/cpcloud/pandas/jobs/8788609...

Just out of curiosity is there any particular reason you don't want to maintain this?

[cpcloud]

scratch that i need to add full deps in python3.3

[cpcloud]

ok upon further inspection 2.6, and the production 2.7 build are working correctly, others are not

2.7 local-override build is compilling something against np 1.7, i need to do further investigation
3.2 is installing numpy1.6.1 from pypi
3.3 i just corrected

[cpcloud]

looks like the versions of pytables and matplotlib used by the local-override build are compiled against np1.7

[cpcloud]

@y-p i see. will change the urls to match the folders on the server

[cpcloud]

@y-p can u throw up a numpy version in /3.2 (1.6.1) and /3.3 (1.7.1)? all the other versions are correctly downloading wheels and no abi incompatibility issues. thanks for all the help!

[ghost]

I put up some new packages on the server, one directory per each.
Having trouble compiling numpy 1.6.1 on 3.2 though. looking through it.

as for maintainting it: just lack of time, nothing else.

[cpcloud]

yeah i got it working except for 3.2 and 3.3

[ghost]

The packages are up on the server, one dir per job, I had to switch to numpy
1.6.2 for py3.2.
I believe the culprit for the 1.7.1 overwriting 1.6.x is patsy (when using pip -I), not only scikits.
installing it with --no-deps seems to do the trick, but that won't work when installing from requirements.txt.
pip -I is needed for forcing existing versions (travis preinstalls certain packages), but can
be dropped if conflicting versions are uninstalled beforehand.

Probably won't have more time to spend on this during the week, so I've disabled the use of
binary packages in master, until all the kinks are sorted out.
@cpcloud if you'd like to push an updated install.sh that makes use of the packages,
please do. If not, I'll get back to this when I can.

[cpcloud]

will do. it's running on travis now...this is the last thing to fix before cutting a release candidate...

[cpcloud]

@y-p would you mind updating ci/speedback/build.sh with whatever you used to make the latest set of packages? i've set up an openshift server, just in case, you know, buses and such. I would be eternally grateful.

[ghost]

glad to hear it, go ahead and move the pointer over to your server.

Here's the script I used:

#!/bin/bash

set -x
export PYTHONIOENCODING='utf-8'

for reqfile in $(ls -1 requirements-*.*); do
    TAG=$(echo $reqfile |  grep -Po "(\d\.?[\d\-](_\w+)?)")
    WHEELHOUSE=$TAG
    PY_VER=${TAG:0:3}
    PIP_ARGS="--use-wheel --find-links=$WHEELHOUSE  --download-cache /tmp"
    echo $reqfile
    rm -Rf /tmp/venv
    virtualenv -p python$PY_VER /tmp/venv
    source /tmp/venv/bin/activate
    pip install -I --download-cache /tmp git+https://github.com/pypa/pip@42102e9d#egg=pip
    # pip install -I --download-cache  /tmp https://bitbucket.org/pypa/setuptools/downloads/setuptools-0.8b6.tar.gz
    # pip install -I --download-cache /tmp distribute==0.6.35
    python /tmp/distribute-0.6.35/setup.py install
    pip install -I --download-cache /tmp wheel

    mkdir -p $WHEELHOUSE
    cp $reqfile $WHEELHOUSE
    cat $reqfile | while read N; do
        echo $N;
        pip wheel $PIP_ARGS --wheel-dir=$WHEELHOUSE $N;
        pip install $PIP_ARGS --no-index $N;
    done
done

the beta pip/setuptools had some issues, had to intervene manually to get things going
on 3.x. YMMV.

here are the requirements filenames:

-rw-r--r--  1 user1 user1    61 Jul  7 03:21 requirements-2.6.txt
-rw-r--r--  1 user1 user1   214 Jul  7 03:22 requirements-2.7_LOCALE.txt
-rw-r--r--  1 user1 user1   327 Jul  7 03:21 requirements-2.7.txt
-rw-r--r--  1 user1 user1   179 Jul  7 03:21 requirements-3.2.txt
-rw-r--r--  1 user1 user1   179 Jul  7 03:21 requirements-3.3.txt

To be placed in ci/speedpack.
The script generates one directory per, which needs to be uploaded to the webserver.

It's your baby now, good luck.

[cpcloud]

thanks.