arm64 CI takes too much time to complete .

[ossdev07]

Hi, I am trying to add support for arm64 in travis CI for Pandas. but unfortunately, all tests are taking more than 50min to complete and hence, travis shows up :
The job exceeded the maximum time limit for jobs, and has been terminated.

I have attached the logs here:
travis-ci-test.txt

Can anyone help me to identify the root cause for this ?

[jreback]

what is the setup time (image + conda install)
what is build time for pandas

i would start with less deps, eg look at the numpy-dev build that only installs minimal

and then go from there

[ossdev07]

@jreback
setup_env.sh takes a total of 855 seconds .
Also, i can't find a minimal distribution for numpy . Can you point me to it?
Do note that miniconda3 does not currently support arm64, so i am using archiconda (which supports arm64) .

[jbrockmendel]

Also, i can't find a minimal distribution for numpy . Can you point me to it?

I'm pretty sure hes referring to this build https://github.com/pandas-dev/pandas/blob/master/ci/deps/azure-37-numpydev.yaml

[alimcmaster1]

Before you spend too much time on this - are we still wanting to move away travis? To GitHub Actions @datapythonista

[datapythonista]

I don't think github actions supports arm64. If we have arm64 users and we want to have it in the CI I guess it makes sense. Does numpy have arm64? I don't think we should have it if it doesn't, the CI of pandas is already huge and slow, and I guess we don't want to overtest with platforms not explicitly supported by the rest of the ecosystem.

[jreback]

since conda has builds for arm64 having a CI build makes sense -
we don’t have explicit support at all but would take patches

[datapythonista]

does it make sense to replace the architecture of one of our current builds, instead of adding a new one?

[jreback]

not until we have a working build

[ossdev07]

since conda has builds for arm64 having a CI build makes sense -
we don’t have explicit support at all but would take patches

@jreback are you reffering to archiconda (arm64 alternative for miniconda)?
https://github.com/Archiconda/build-tools/releases

[ossdev07]

Looks like, for pandas, 50 minutes is not enough for even to start the tests sometimes . For reference, here is the link for build in travis-ci:
https://travis-ci.org/ossdev07/pandas/jobs/625636386

I am estimating it would take approx 30 minutes more to complete i.e. 1 hour 20 minutes .
The issue might be due to absence of wheels for numpy on arm64 as it is not cached .
https://travis-ci.community/t/no-cache-support-on-arm64/5416/12

can anyone help resolving this issue?

[jreback]

you might want to try this on azure instead
iirc we have build timeouts there and likely could get that if needed

[michal-at-travisci]

Hi @ossdev07
I left you a direct message on Travis CI community forum. Long story short - you should be able to build now pandas repository with longer timeouts, which should enable your arm builds with Travis CI. Please take it for a spin and see if it works for you. Also the hints provided in Travis CI community forum may help speeding up the builds.

As a side note, I'd like to point you to a couple of Travis CI documents:

1. Common Build Problems - My Builds Are Timing Out
2. Build Stages
3. Caching
4. Using Workspaces

Maybe some of these techniques could be used to speed up the pandas builds & tests? Usually splitting work into build stages and caching does a lot of improvement (if it's only applicable for your scenario and you decide to use Travis CI for your builds & tests).

Happy building and a happy New Year!

[jreback]

canceling in favor of #36719