partest should implement --failed correctly

[odersky]

It would be really helpful to have a mode where partest runs only failed tests.

[odersky]

More items on the partest wishlist:

• Provide a log per test, instead of piping everything into one output stream
• Honor xxx.flags files when performing junit tests
• Have a cleaned up output like the original partest

[DarkDimius]

Honor xxx.flags files when performing junit tests

AFAIK this is implemented. Flags are generated from junit config.
Otherwise many tests requiring -language:Scala2 would fail.

[odersky]

It would be good to have a test for this. I tried something with -language:Scala2 and it did not work.

[DarkDimius]

We do have tests for this. For example compileStdLib test https://scala-ci.typesafe.com/job/dotty-master-validate-partest/708/consoleFull .
Here the excerpt from log:

/home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/compileStdLib/Signalling.scala:89: 
migration warning: Procedure syntax no longer supported; `=' should be inserted here
 def setIndexFlagIfGreater(f: Int)
 ^

This would have been an error if partest did not pass -language:Scala2

[odersky]

Ah, then it was probably the junit test that failed. But given that junit cannot honor the flags, maybe we should stop using it? I would switch tomorrow if partest's test output was acceptably nice.

[DarkDimius]

@odersky junit also honors the flags, otherwise the same tests would fail in junit.

I'd like to continue with being able to choose to run tests either in partest or junit.
My workflow is based around attaching debugger to junit tests.

[smarter]

junit also honors the flags, otherwise the same tests would fail in junit.

Are you sure? Doesn't junit work because we explicitly pass the scala2 mode flag in tests.scala:

@Test def compileStdLib = compileList("compileStdLib", stdlibFiles, "-migration" :: scala2mode)

[DarkDimius]

@smarter partest flags are generated from ones that you explicitly pass to junit. Not the reverse.

[smarter]

Partest is dead, but let's open an issue for having this feature in vulpix: #3798