CI: split Windows Azure tests in half

[Dr-Irv]

Trying an experiment for Windows CI where PYTEST_WORKERS is auto rather than 2.

[jreback]

how'd this work out?

[Dr-Irv]

how'd this work out?

auto doesn't solve the problem. Current hypothesis is that the expression for skipping tests is wrong. Here's the evidence:

For the Windows py38 tests, which are "not slow and not network", we get: 122956 passed, 6965 skipped
For the Windows py39 tests, which are ""not slow and not network and not high_memory"", we get 124788 passed, 4943 skipped

Doesn't make sense that skipping more tests in the expression would yield less tests skipped.

That test is running now....

The other issue is that it isn't clear to me how many cores are actually assigned to the Azure instances. Using auto means use as many cores as are available. In the first set of tests under this PR, py38 and py39 both used two cores. In the latest push, py38 is using 4 cores and py39 is using 2. The more cores we can get, the faster these tests will run.

[jreback]

38 and 39 might have different deps though

[mzeitlin11]

In the latest push, py38 is using 4 cores and py39 is using 2. The more cores we can get, the faster these tests will run.

Just noting from looking at this in #42236, seemed like the opposite was actually true based on looking at a few runs using different numbers of cores.

[Dr-Irv]

Revised hypothesis. The number of cores that we get from Azure is variable. But the issue here is that if you only have 2 cores and use both of them, performance can get worse than if you use just 1 core. Here is some evidence. On my 16 core laptop, I ran pytest --skip-slow -m "not single" -n XX pandas/tests/groupby for XX=1,2,4,8,16 and auto. Here are the timing results:

XX	Time (secs)
16	32.09
8	30.16
4	37.56
2	55.04
1	90.53
auto	28.87

For "auto", it created 8 processes.

Note that using all 16 cores doesn't help - that's because these are hyperthreaded cores, and using them usually doesn't help computational performance. Secondly, even going from 1 core to 8 only sped things up by a factor of 3.

So if Azure gives us 2 cores, and we use both, that could be worse than just using 1 core. My latest commit is doing that, and also (hopefully) printing out the CPU info.

[jreback]

oh so this really could be a timeout?

also is there a way to request 4 cores (min) from azure for these builds?

[Dr-Irv]

oh so this really could be a timeout?

Yes. The last test confirms it. Here is the processor info that was obtained (which was the same for the py38 and py39 runs):

Caption                            DeviceID  MaxClockSpeed  Name                                            NumberOfCores  

Intel64 Family 6 Model 85 Stepping 7  CPU0      2594           Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz  2              

In the last test, I had it not do any parallelization, and the py38 run timed out, the py39 run did not. So I think this has to do with where the virtual container is sitting on the hardware. Most of the time before, I'd see py38 succeed and py39 time out.

also is there a way to request 4 cores (min) from azure for these builds?

I don't know the answer to that! Who is responsible for the relationship we have with azure?

[jreback]

I don't know the answer to that! Who is responsible for the relationship we have with azure?

umm, don't really have anything specific

[Dr-Irv]

Another possible solution is to split the tests for Azure. Here's an example from Microsoft on how to do that:
https://github.com/PBoraMSFT/ParallelTestingSample-Python

[jreback]

maybe best is just to remove some tests (e.g. have a marker that we dont' use here) and/or move some to a slow build.

[Dr-Irv]

maybe best is just to remove some tests (e.g. have a marker that we dont' use here) and/or move some to a slow build.

Issue here is that the slowest test is 10 seconds. Our issue is volume (over 124K tests) and on Azure, we're running on a "slow" machine compared to Travis.

[jreback]

maybe best is just to remove some tests (e.g. have a marker that we dont' use here) and/or move some to a slow build.

Issue here is that the slowest test is 10 seconds. Our issue is volume (over 124K tests) and on Azure, we're running on a "slow" machine compared to Travis.

right one way is for example to NOT run any pandas/tests/windows (which @mroeschke disabled for a while)
not great, but we do really test these

[jbrockmendel]

If we cant get more cores/build, could we run more azure builds, e.g. for each of the current azure builds split it in two with one ignoring tests/window and the other doing only tests/window?

[mroeschke]

Or maybe we can test a separate Github Actions Window runner for a subset of tests (seems to only be a 2 core CPU https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources)

[Dr-Irv]

I'm trying something else here by setting up a PYTEST_TARGET in the azure pipelines and then for Windows having separate jobs where I separate the tests into "pandas/tests/[a-i]*" and "pandas/tests/[j-z]*"

[jreback]

looking good

can u close and reopen to force a run again (just checking on the timeouts)

[Dr-Irv]

can u close and reopen to force a run again (just checking on the timeouts)

Closing and reopening..

[jreback]

@Dr-Irv this looks good. i think splitting the macos tests along the same lines makes sense as well (in this PR or followon)

[jreback]

@simonjayhawkins lmk if you want to backport (either way ok by me)

[Dr-Irv]

@jreback all green

We also have the option of speeding up some of the other checks by doing a similar split of the tests. Some of them are taking close to an hour. That would allow people to get quicker feedback on their PRs. If you think that's a good idea, I could do another PR for that.

[jreback]

@jreback all green

We also have the option of speeding up some of the other checks by doing a similar split of the tests. Some of them are taking close to an hour. That would allow people to get quicker feedback on their PRs. If you think that's a good idea, I could do another PR for that.

yep, we actually have a pretty large azure allocation, so a few more jobs per PR is fine.

[jreback]

going to backport this (@simonjayhawkins can always not merge it on 1.3 if too many conflicts)

[jreback]

thanks @Dr-Irv

[jreback]

@meeseeksdev backport 1.3.x

[jreback]

@Dr-Irv if you can push the backport as above (guess we have a conflict)

[Dr-Irv]

@Dr-Irv if you can push the backport as above (guess we have a conflict)

Created this PR #43496

Not sure which labels and milestones to put on it, and the backport PR is showing conflicts

[Dr-Irv]

@jreback Bad PR #43496 (was merging to master). New PR #43497 merges to 1.3.x .

[jbrockmendel]

Awesome, thanks for figuring this out @Dr-Irv