Partest for dotty

[vsalvis]

Use partest to run pos and neg tests that were run by test/dotc/tests.scala before.

Compiler flags can be provided per directory by a __defaultFlags.flags files, or per test through testname.flags (in this case, __defaultFlags.flags is ignored). For neg tests, the expected number of errors can be provided in a testname.nerr file.

[DarkDimius]

I have very mixed feelings about this PR.
Testing is the way how I organize my work: I isolate test that fails, add breakpoints and run single test with debugger attached.

It feels that by using partest for pos tests we loose a lot of flexibility and clearness. Let me illustrate this on several examples:

• Currently tests.scala can be used to see which tests succeed till the very backend. With partest it seems that directory structure needs to be used for this. I would argue that a single file is a lot easier to follow than filesystem tree hierarchy to understand what is happening.
• there's an IDE support for current setup, ie if one wants to run a single tests IDE helps with this, or if one wants to see what tests are disabled due to failure in ElimByName there's a variable to track with IDE failedbyName.
• imagine that I want to run a single test twice but with different flags. I used to do if when working on pattern matcher. As directory structure here defines it, it seems that I would need to copy test twice and make sure both copies are in sync.

I do not have enough experience with partest, so maybe those concerns are caused by lack of skill of using it. Though I assume I'm not only one, who has more experience with JUnit or ScalaTest-like frameworks.

Based on this, I'd preffer to leave pos tests under JUnit, or be able to share them between JUnit and ParTest. Also note that JUnit is also intergated with our benchmarking infrastructure(yes, we have it running, check this out: http://goo.gl/k26LY2): #156 so this PR breaks it. I wonder if similar integration can be done with ParTest. Could it be adapted to discover JUnit test definitions and run them?

A sidenote: travis uses 1.5-core machines. There wouldn't be much win from using partest there.

[DarkDimius]

From high level view. In perfect world we want to have infrastructure that verifies:

• tests pass(both pos, run and neg)
• performance doesn't regress
• code coverage tests

Currently this all is done in quite a compact manner, where single file that defines tests. Performance regression is tested for them automatically. Code coverage is also easy to add: I'm actually use scoverage on my machine to see which test executes some particular line.

By using Scala to define tests we allow to reuse flags via variables, add default flags via emplicits, and a lot more.

I believe that having this all defined by directory structure is too limiting, and we'll need to reinvent a wheel.

What I propose instead, is to have tests defined in the way they are defined now, but allow ParTest to be a JUnit runner, and generate directory structure for it(including check files) in runtime.
This will allow to use either standard JUnit runner(or the ones that Eclipse and Intellij define for intergration into IDE), or use ParTest when there's need for it.

[odersky]

Let's discuss next week when I am back. I think we probably will need to run the current tests and partest in parallel, but we can try to avoid duplication, by not having the same test in both frameworks. Partest has the advantage that it makes it far easier to move tests over from scalac, and that is important right now. But I would not want to give up all of the flexibility of the current testing setup.

[vsalvis]

Closing this pull-request as we've decided on a different path, namely:

• keeping the tests.scala file which defines tests using JUnit annotations, with the full convenience of defining flags programmatically
• adding partest as a JUnit runner that generates the necessary flags files