Need another escape hatch for experimental

[odersky]

@experimental definitions and experimental language imports can currently only be used on code that is itself @experimental or that is compiled with a nightly version of the compiler.

This effectively locks experimental away in the poison cabinet.

For instance, I see no practical scheme to use experimental features in the compiler itself. This is a shame. First, there is no issue in using experimental features in the compiler. If they change, we can change the compiler as well, it's the same repo. So the stability argument does not apply. Second, we forgo the large benefits of dog-fooding, which means that experimental features are pushed out with much less testing and user experience than would have been the case otherwise.

To change that, I propose we create an setting -Yexperimental which would make the compiler pretend it is a nightly version. I am not sure what this should mean for the Tasty version, however.

[bishabosha]

Perhaps we can check that the source file of the code is in a git repo matching the dotty repository?

[odersky]

That's one option. But it means we still put up very high hurdles for everyone else.

[smarter]

-Yexperimental could act as if all classes and traits were defined with @experimental.

[smarter]

... one issue there is that part of metals depend on the compiler and would need to be experimental too, so this would make it more important to have a stable API for the presentation compiler that metals can call into.

[prolativ]

This idea seems familiar #14306 (comment)

[sjrd]

part of metals depend on the compiler

As well as every compiler plugin out there, for example Scala Native.

[sjrd]

Also tooling such as Metals will not support these things. So if you do use them in the compiler, you cannot expect Metals to work for the compiler codebase anymore. Nor any IDE, for that matter.

[odersky]

Also tooling such as Metals will not support these things. So if you do use them in the compiler, you cannot expect Metals to work for the compiler codebase anymore. Nor any IDE, for that matter.

Not necessarily. All I want is refer to some classes that are marked @experimental. I don't see why Metals would stop working on the compiler codebase that way.

If we do introduce experimental language imports then it's different. It depends what they are. If they affect syntax it's possible that IDEs would not longer recognize this correctly. But that's not a complete show-stopper either. I have been working with occasional red underlines from metals for a long time previously.

If we label the compiler transitively as experimental then I agree it won't work. We need to have a way to override experimental rules locally without the codebase becoming itself experimental.

[odersky]

So, the radical option would indeed be to compile the compiler with -Yexperimental, which would force all plugins and other code using the compiler to be compiled with -Yexperimental as well. Actually, for metals it would only be those parts that rely on compiler internals, but it would take some work to factor them out cleanly.

In a sense that's just being honest. Internal compiler APIs are not stable, and can be changed at any time, which is our definition of what experimental is. Maybe that would push the compiler maintainers to develop richer stable APIs and other tools to use these APIs instead of compiler internals. But that's a process that requires buy in from many parties.

In summary I see the following options:

• Add -Yexperimental in the sense that every compiled unit is treated as if it was annotated @experimental. That allows applications to use experimental features but it's not an attractive options for libraries or the compiler itself (since that one is also a library that others use).
• Have a special exclude that allows the compiler to access experimental features without itself being experimental. The idea is that if an experimental feature changes, the compiler will change as well. Since everybody using the compiler needs to be prepared for any changes in the compiler, this is OK.
• Make that special exclude also available to others in some way.

What do you think? Which (combinations of) options should we go for?

[soronpo]

In any case, I think this should be brought up as a SIP for a committee vote.

[makkarpov]

-Yexperimental could act as if all classes and traits were defined with @experimental.

-Yskip:crossVersionChecks just acts as if @experimental has never existed.

[tgodzik]

Have a special exclude that allows the compiler to access experimental features without itself being experimental. The idea is that if an experimental feature changes, the compiler will change as well.

How does it work with tasty? Wouldn't it be a possibility that the tasty changes and breaks all the plugins downstream which compile with stable versions? We would need a tooling community build to test that out or Mima for Tasty?

Since everybody using the compiler needs to be prepared for any changes in the compiler, this is OK.

Including all the tooling authors? This works when we have a proper API to use, otherwise we should be careful about big changes that can impact tooling especially using experimental.

This effectively locks experimental away in the poison cabinet.

Isn't it exactly what experimental is supposed to be? We don't want to use experimental in a real live code since it's something we are not yet sure about. And people might have less confidence in the compiler if it uses experimental features.

Overall, this change seems a bit hostile for tooling and I would try to keep in my mind that tooling authors can't be taken for granted. Aside from Metals of course 😅

[nicolasstucki]

@odersky I would like to understand what kind of experimental features we would like to access from the compiler.

I had the use case of accessing the MacroAnnotation. For this case, I just used plain JVM reflection. Then to make it nicer, I used reflectiveSelectable and created a mock interface that allowed me to call the experimental methods as if they were present.

import scala.reflect.Selectable.reflectiveSelectable
type MacroAnnotation = {
  def transform(using Quotes)(tree: Object/*Erased type of quotes.refelct.Definition*/): List[MemberDef /*quotes.refelct.Definition known to be MemberDef in QuotesImpl*/]
}
val annotInstance = ???.asInstanceOf[MacroAnnotation]
annotInstance.transform(using quotes)(tree)

Does this approach also work for your other use cases?

[odersky]

I think we should not jump through hoops with reflection here. We need to use something that's experimental, we should be able to get it without playing tricks like this.