speedup: don't fork a new JVM for tests that don't actually need one

[SethTisue]

Each run (and jvm) test always forks a fresh JVM; this costs a lot in startup time, and we're paying that cost about 1800 times. @retronym observes that tests that actually need this level of isolation are probably relatively few. Most of the time we should be enough to stay in partest's own JVM and just isolate each test in a fresh classloader. We'll need to capture stdout and stderr and trap System.exit.

@vpetro, @dwijnand, @retronym and I wrote wip code for this today at a Lightbend engineering retreat. Our preliminary results indicate an average overall speedup of about 2x. If that holds, this will substantially speed up PR validation.

PR to follow. It may take a while to identify tests that need the old level of isolation. (We've already seen cases where test A does something later causing test B to fail, so it isn't a trivial matter of just running any failed test the old way.)

[SethTisue]

in a similar vein, see also the new ticket scala/scala-dev#295, which proposes removing a lot of tests from partest's clutches entirely

(not to mention the recently merged scala/scala#5661, which removed our ScalaCheck-based tests from partest's clutches)

[SethTisue]

example failure causes to watch out for:

• SI-5941 makes printing of type names sensitive to classloader details
• use of security manager calls
• calls to System.exit (Dale is working on it)
• altering of system properties (Dale added code that saves & restores them)
• @retronym says using Scala reflection may alter the state of the VM in ways that may influence later tests
• ...?

[SethTisue]

very rough numbers: in our local tests so far this saves about 2/3 of a second per run test. if we can enable this for 1500 (out of 1800) tests, we'll save 1000 seconds = 16 minutes. PR validation currently takes 71 minutes total.

[retronym]

Some progress towards fixing SI-5941: https://github.com/retronym/scala/tree/ticket/5941

[SethTisue]

so it isn't a trivial matter of just running any failed test the old way

I think Dale convinced me that wouldn't be so bad as a starting point. When test A does something that later causes test B to fail, it's hard to say who's at fault. Is it test A for changing something, or test B for depending on something that could change? So yes, mass quarantine of all the test Bs will catch some innocent victims, but only some.

[dwijnand]

I've not forgotten this. It's on my todo list (of out-of-hours todos).

[SethTisue]

It's still very much on my mind as well.

[retronym]

To rekindle this, I've created a branch of scala/scala that uses partest-as-a-library from a JUnit suite, overriding the way that run tests are executed as discussed here.

I wrote this from scratch as I can't remember what we did with the code we wrote in Banff: system property setup, capturing stdout/stderr, trapping sys.exit.

https://github.com/retronym/scala/tree/topic/jptest

18 minutes later:

[info] Test run finished: 37 failed, 0 ignored, 1872 total, 1051.299s
[error] Failed: Total 1872, Failed 37, Errors 0, Passed 1835
[error] Failed tests:
[error] 	partest.RunTests
[error] (junit/test:testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1057 s, completed 14/02/2018 6:36:51 PM

Running all 1877 partests under the existing sbt 'partest --run' is glacially slow -- 71 mins -- on the same machine:

[error] Failed: Total 1877, Failed 3, Errors 0, Passed 1874
[error] Failed tests:
[error] 	partest
[error] (test/it:testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 4255 s, completed 14/02/2018 7:50:01 PM

... and that's with use of all four cores!

Maybe something is off with my measurements, but they suggest an even greater improvement than the 2x we got from the back-of-the-envelope calcs above.

[retronym]

One advantage of using JUnit as the harness is that it we get some tooling for free. For example, running this suite in IntelliJ, I can sort by test duration.

I'm not sure why the IntelliJ runner took 29min in total, rather than the 17min I measured through SBT. But in that context, the slowest 10 tests account for > 30% of the runtime.

Some of these tests are doing black box testing for absence of memory leaks in ways that stress GC (t6969, reflection-mem-tests*). Others are performing stress tests on collections (t6853, ctries-new, ctries-old, t3989).

Test of reify tend to be slow-ish, and there are lots of them (135).

Tests that themselves perform another layer of programmatic compilation (through Toolbox.eval or through ReplTest) are also sluggish.

At the other end of the scale, the fastest executing tests still take around 130ms. This is likely limited by the time taken to do a hello world compile with a new Global, even with a warmed up JVM.