Remove legacy tests #3962
Remove legacy tests #3962
Conversation
This new test checks that the framework rejects incorrect pos/neg and run test. I think addition meta tests should rather be added as part of the unit test infrastructure, given than this is rather heavy to maintain. (Note that sbt-output.check is manually annotated with "SKIP" lines)
OlivierBlanvillain
force-pushed
the
no-legacy-no-more
branch
from
544c08f to
a4f9e2d
Compare
| * These are legacy, do not add tests here, see `CompilationTests.scala` | ||
| */ | ||
| @Category(Array(classOf[LegacyTests])) | ||
| class tests extends CompilerTest { |
There was a problem hiding this comment.
I believe you can now remove CompilerTest
project/scripts/cmdTests
Outdated
|
|
||
| echo "running Vulpix meta test" | ||
| tmp=$(mktemp) | ||
| if sbt "dotty-compiler/testOnly dotty.tools.vulpix.VulpixMetaTests" > "$tmp" 2>&1; then |
But we could just run vulpix outside of sbt after compiling it to avoid this problem. |
|
@smarter Still, these end-to-end tests require manual annotation of the text output of vulpix (see the lines annotated with SKIP at EOL in this file). It's just more convenient to write a regular unit test, for instance this commit 8e0f219 took around 5 minutes to write, I don't see what's to be gained by turning that into an end-to-end test. If we know that a failing run test throws an exception that makes the CI fail (the meta test), and that a timing out run test throw an exception, then it should be safe to deduce that a timing out run test makes the CI fail. |
OlivierBlanvillain
force-pushed
the
no-legacy-no-more
branch
from
fe81adf to
1d77144
Compare
Before that sbt ran without the -Dsbt.ivy.home option set and the dependencies where downloaded again...
OlivierBlanvillain
force-pushed
the
no-legacy-no-more
branch
from
1d77144 to
e2dd138
Compare
| @Test def runTimeout: Unit = { | ||
| try { | ||
| compileFile("tests/vulpix-tests/unit/timeout.scala", defaultOptions).checkRuns() | ||
| assert(false, "unreachable") |
There was a problem hiding this comment.
fail("unreachable")| compileFile("tests/vulpix-tests/unit/deadlock.scala", defaultOptions).expectFailure.checkRuns() | ||
|
|
||
| @Test def badJava: Unit = | ||
| try compileFile("tests/vulpix-tests/unit/BadJava.java", defaultOptions).suppressAllOutput.checkCompile() |
There was a problem hiding this comment.
try {
....
fail("unreachable")
}| @Test def badJava: Unit = | ||
| try compileFile("tests/vulpix-tests/unit/BadJava.java", defaultOptions).suppressAllOutput.checkCompile() | ||
| catch { | ||
| case ae: AssertionError => assert(ae.getMessage.contains("java compilation failed")) |
| assert(false, "unreachable") | ||
| } catch { | ||
| case ae: AssertionError => | ||
| assert(ae.getMessage == "Run test failed, but should not, reasons:\n - test 'tests/vulpix-tests/unit/timeout.scala' timed out") |
There was a problem hiding this comment.
assertEquals(ae.getMessage, ...)
project/scripts/cmdTests
Outdated
| fi | ||
| tmp1=$(mktemp) | ||
| cat "$tmp" | sed '/Test run started/,$!d' > "$tmp1" | ||
| while read expected <&4 && read actual <&3; do |
There was a problem hiding this comment.
I think I would remove the -x option of the script. Otherwise the code below prints a lof of noise
| } | ||
| catch { | ||
| case _: IllegalArgumentException => // pass! | ||
| case NonFatal(_) => fail("wrong exception thrown") |
There was a problem hiding this comment.
I would remove this line
| fail("didn't fail properly") | ||
| } | ||
| catch { | ||
| throw new Exception("didn't fail properly") |
There was a problem hiding this comment.
Maybe, factor this into a method fail() with a comment saying that JUnit Assert.fail cannot be used because it throws an AssertionError and that's what we expect to catch.
OlivierBlanvillain
force-pushed
the
no-legacy-no-more
branch
from
1e3135c to
cf8bbb3
Compare
This PR removes the legacy tests infrastructure and adds a simple meta test to check the output of Vulpix. I don't think it makes sense to test all features of Vulpix using this style given that the output of sbt is quite unreliable (timing annotations and out of order execution).