Tests need to allow experimental imports

[odersky]

The situation so far was that we have a menagerie of Y flags that enabled non-standard feature and and associated menagerie of meandering test scripts that enabled these flags for specific tests.

Language imports promised to change all that. Instead of having a test in some magic location or with some magic invocation, we specify the assumptions of the test in the test sourcecode itself. This works great, and would be an ideal mechanism to try out future language features in tests.

Unfortunately we shot ourselves in the foot with #11852. Now, experimental imports are disabled for normal tests. What you have to do is

• add the language import
• put the test in some special directory where all experimental features are enabled
• adapt the test script to special treat this directory. If we decide we need an experimental import for a new class of tests, we need to create the directory, put the new test in it, and change the test runner script to recognize that directory.

I find that bureaucracy unbearable. It will lead to everyone reverting to special flags for experimental features which conveniently avoid the language import overhead. And that in turn will lead to experimental features that are completely unchecked, since only language imports are checked but not flags.

So, in summary #11852 achieves exactly the opposite of what it was supposed to achieve: It makes dealing with experimental features so burdensome,that everybody will take evasive action.

I think we need to solve this before we go into RC2.

[liufengyun]

adapt the test script to special treat this directory. If we decide we need an experimental import for a new class of tests, we need to create the directory, put the new test in it, and change the test runner script to recognize that directory.

This is not needed, just put the test in one of the folders is enough:

• tests/neg-custom-args/experimental
• tests/pos-custom-args/experimental
• tests/run-custom-args/experimental

We cannot just use import for an experimental feature in the tests, as the tests will also be compiled by a RELEASE compiler, which does not support experimental features.

[odersky]

This is not needed, just put the test in one of the folders is enough:

Yes, and I think that's completely unacceptable, as I explained in the issue.

[liufengyun]

This is not needed, just put the test in one of the folders is enough:

Yes, and I think that's completely unacceptable, as I explained in the issue.

I see two alternatives:

• to support experimental features in RELEASE compilers under a flag (still need to enable the flag when adding a test).
• revert Fix #11774: only enable experimental features for snapshot and nightly #11852 and abandon the idea of only support experimental features in nightly.

[odersky]

to support experimental features in RELEASE compilers under a flag (still need to enable the flag when adding a test).

I think that's what we have to do. Or use a private settable variable that is set only in the test runners.

[odersky]

In any case we have to get rid of those experimental subdirectories. That's the most urgent thing to do. That was a lot of change for getting a much deteriorated workflow.

[liufengyun]

In any case we have to get rid of those experimental subdirectories. That's the most urgent thing to do. That was a lot of change for getting a much deteriorated workflow.

It is now reverted: #11916.

[odersky]

Thanks for the quick reaction @liufengyun! We should still disable experimental imports in stable releases but with less drastic changes

[liufengyun]

We should still disable experimental imports in stable releases but with less drastic changes

The goal described above is to treat an experimental test as a normal test (to easily put it anywhere). It implies the stable release compiler has to support experimental features (via a compiler flag or other hacks). And we still need another flag to disable experimental features in NIGHTLY compiler for neg tests of experimental language imports.

So each way is a trade-off: either we follow the convention to put experimental tests in appropriate folders, or we compromise the guarantee that users cannot use experimental features in stable releases.

[odersky]

The enabling flag could be a private variable that is set to true only from within our internal testing framework. That means we allow experimental in our internal tests but not in other code compiled by a release compiler.