Allow customisation of metaprogramming via scalac flags

[japgolly]

This issue is to allow inline code to customise its output based on user-defined settings provided as scalac flags.

Background

Full detail here: https://contributors.scala-lang.org/t/metaprogramming-configurability/4961

This issue is the first part of the solution I proposed above.

Proposal

Disclaimer: the names used below are just drafts and still need some good bikeshedding.

• Add a new scalac flag that takes a -E:key=value (similar to how java accepts -Dkey=value args)
• Add transparent inline def envGet(inline key: String): Option[String] to scala.compiletime
  that provides access to the settings specified with above scalac flags

Usage Example

The very first example in the inlining doc is a logger that can be configured at compile-time, and is "zero-cost" (as they say) at runtime.

• when logging = true then calls to log() generate println statements
• when logging = false then calls to log() generate nothing
• there is never a logging-config check at runtime

Currently, the problems with this are

1. Logger and its Config are static, and changing the setting is a code change
2. Config must be pre-configured and defined alongside Logger (else Logger wouldn't compile)
3. If Logger were a library, downstream users would have no way of configuring it

If this issue were implemented, our zero-cost-ish Logger could be written like this below, to accept a "myLogger.level" setting that downstream users can populate:

import scala.compiletime.*

object Logging {
  private inline val Trace = 0
  private inline val Debug = 1
  private inline val Info  = 2
  private inline val Warn  = 3

  private transparent inline def chosenThreshold: Int =
    inline envGet("myLogger.level") match
      case Some("TRACE") => Trace
      case Some("DEBUG") => Debug
      case Some("INFO")  => Info
      case Some("WARN")  => Warn
      case None          => Warn // let's provide a default out-of-the-box
      case Some(x)       => error("Unsupported logging level: " + x)

  private inline def log(inline lvl: Int, inline msg: String): Unit =
    inline if lvl >= chosenThreshold then println(msg) else ()

  // This is the public API
  inline def trace(inline msg: String) = log(Trace, msg)
  inline def debug(inline msg: String) = log(Debug, msg)
  inline def info (inline msg: String) = log(Info , msg)
  inline def warn (inline msg: String) = log(Warn , msg)
}

And then a downstream user could specify -E:myLogger.level=INFO in their scalac flags so that our toy logging library effectively becomes this for them:

object Logging {
  inline def trace(inline msg: String) = ()
  inline def debug(inline msg: String) = ()
  inline def info (inline msg: String) = println(msg)
  inline def warn (inline msg: String) = println(msg)
}

Reconsidering the three problems above, with this new solution:

1. Logger can now be configured dynamically, and via config (no code changes required)
2. Config is no longer necessary. Whether the library author wants to provide it and/or defaults is now something under their control.
3. Downstream users can now configure the behaviour of Logger

PR?

I'm not asking that the busy Scala 3 team take some time out to implement this. This seemed simple enough for me to try implementing it myself and so I have, and it seems to work well! The above logging example is one of the tests and is confirmed to work as excepted.

[nicolasstucki]

This looks like overkill. Wouldn't it be simpler to just use java properties and simply define a macro that reads them?

import scala.quoted.*

inline def envGet(inline key: String): Option[String] = 
  ${ getLogLevelExpr(key) }

def envGetExpr(key: Expr[String])(using Quotes): Expr[Option[String]] =
  Expr(scala.util.Properties.envOrNone(key.valueOrError))

This does not need to be in the standard library.

[nicolasstucki]

If we really need a compiler flag for the environment, then it this should be exposed in reflect and compiletime should not have any extra functionality.

[nicolasstucki]

⚠️ Note that this is an extremely dangerous feature as it will introduce instability in the generated artifacts long term. Once we allow recompilation from TASTy, having a different set of flags could change the behavior of the already distributed artifact. Each user might see different behaviors depending on their setup. Additionally, this change of behavior would be different in transparent and non-transparent inlining. ⚠️

[nicolasstucki]

This is also incompatible with incremental compilation.

[nicolasstucki]

BTW, it is possible to implement such a logger without the need of an external flag. We have already done that in dotty.tools.dotc.config.Printers.

[japgolly]

I initially had that thought too but it doesn't work. Let me get into the details:

Why not use java properties?

1. They can't be specified from within sbt (see (3) below). They have to be specified outside of sbt.
   • You'd have to start sbt like sbt -Dblah=blah
   • You'd now have most of your build in sbt-land and some that are completely external, which is quite inconsistent and exceptional to common practices and user-expectations
   • IDEs like Intelij and Metals wouldn't be aware of these external settings
2. It would be a global setting that affects all sbt projects. I can already envision cases where I'd want different settings applied to different modules in the same multi-module build.
3. The above two points can't be solved by sbt.
   • Compilation runs from the same sbt process (meaning it gets the process's java props)
   • Having projects try to modify java props before compilation breaks parallelism
   • In order to have different settings for different modules sbt would have to fork / create new processes, which comes with two problems of it's own: 1) starting a new process every time means the JVM would never warm up and compilation speed would be slowed too unreasonably, 2) starting external servers (to keep the forked JVM warm) would be way too much work to implement and greatly increase total memory usage

Incremental compilation

I don't see how this would be incompatible with incremental compilation. Incremental compilation already takes scalac flags into consideration; when certain scalac flags change, certain things are recompiled. Whether that's incremental compilation reacting to the flags, or sbt reacting to the flags and forcing invalidation, the point is that there's already a mechanism in place.

If we used java props then that would *really be incompatible with incremental compilation. If anything, this approach of using scalac flags is what gives us (or can easily be made to give us) correct incremental compilation.

TASTY recompilation differences

• we'd have the same problem using java props - using scalac flags doesn't change this aspect
• customising code generation by a global flag is something that would only be used rarely. In the vast majority of cases, implicit config (or explicit like your printer example) is the way to go. Users who use this approach will have specific needs and it's reasonable to expect them to understand the consequence of relying on compilation-time dynamic global config. The fact that something could be recompiled downstream if downstream settings differ is a feature, not a bug, because this is exactly the problem I ran into with scalajs-react. There's an open bug in scalajs-react because my dynamic global config stuff didn't survive republishing, and so downstream settings were effectively ignored.

[japgolly]

Once we allow recompilation from TASTy, having a different set of flags could change the behavior of the already distributed artifact

@nicolasstucki Not sure if this is already on your radar but @elidable is something else to consider that's in the same bucket.

[nicolasstucki]

Ok, seems that most of my worries have some solutions already.

What would you expect when setting -E:key=value in a normal inline method or in a transparent inline method? How would that interact when you recompile a published artifact that may have been compiled with a -E:key=value2?

[Katrix]

Could the tasty files just not store all the options it was compiled with, bypassing the problem, and providing an official way to do things that is less prone to breaking?

[japgolly]

What would you expect when setting -E:key=value in a normal inline method or in a transparent inline method? How would that interact when you recompile a published artifact that may have been compiled with a -E:key=value2?

@nicolasstucki Sorry I'm not sure what you mean. From the inlining point of view, access to this little env-map would be read-only and the only way to modify it, would be to go to the build tool and change the scalac flags. I'm not quite sure how tasty works but I imagine that the inline methods would always access the env-map as it's defined at expansion time. Let me provide an example below that I hope will be clear...

Could the tasty files just not store all the options it was compiled with, bypassing the problem, and providing an official way to do things that is less prone to breaking?

@Katrix There may be some cases where you would want those options stored in TASTY so that it's repeatable and wont change downstream, but I didn't think about that. In my (limited) real-world scenarios that have motivated this proposal, I explicitly do not want that behaviour though. I want upstream inline code to be re-evaluated downstream according to downstream config (and ignoring upstream's config). Supporting the ability to lock-in/remember certain env settings might be a missing feature in the current version of this proposal. It can be partially-supported already in userland for some cases (see the example below). One way of achieving proper, full support could be to specify alongside the scalac option, that you want the setting remembered and stored in TASTY (eg maybe -EE instead of -E). I don't know if that would really be overkill or not, but a good thing is that we can break this into two steps so that the ability to have TASTY remember certain options can be implemented on top of this first step without much disruption, so it could be added in future when we're more confident there's a need. (Although I assume I'd need much more buy-in from the Scala 3 team to modify TASTY, not to mention a bit of guidance on the implementation).

Example

Imagine:

Library 1:

// compiled without any -E flags

inline def logInDebugMode(inline msg: String): Unit =
  inline envGet("mode") match {
    case "debug" => println(msg)
    case _       => ()
  }

inline def square_v1(n: Int): Int =
  val result = n * n
  logInDebugMode("Calculating $n² = $result")
  result

// As above but not inline
def square_v2(n: Int): Int =
  square_v1(n)

and now a downstream project that depends on Library 1:

// compiled with -E:mode=debug
@main def main =
  val x = square_v1(2)
  val y = square_v2(4)
  println(s"x = $x, y = $y")

I'd expect the output to be:

Calculating 2² = 4
x = 4, y = 16

because:

• square_v1 is inline so is evaluated downstream where -E:mode=debug, therefore it logs
• square_v2 is not inline and therefore has a static body that was compiled in Library 1 where there were no -E flags. The TASTy of square_v2 is effectively the same as n * n without any references to logInDebugMode and therefore, we don't see Calculating 4² = 16 in our output.

So basically the env-map is only considered at the time when inline methods are being reified/spliced. The author of our fake library above has the ability to decide whose -E flags are used by deciding whether a method is inline or not. It's a bit limited but it's certainly enough for the purposes I envisioned, and it's nice and simple.

[japgolly]

I just realised, this could probably be done as a compiler plugin. Maybe I should do that first, let it get some use and then see in the future if there's a good reason to implement this in Scala directly?

If I were to (try to) support retention of flags by storing them in TASTY somehow, is that something that can be done from a compiler plugin?

[nicolasstucki]

I just realized, this could probably be done as a compiler plugin. Maybe I should do that first, let it get some use and then see in the future if there's a good reason to implement this in Scala directly?

A compiler plugin sounds like a good way to start.

If I were to (try to) support retention of flags by storing them in TASTY somehow, is that something that can be done from a compiler plugin?

One way could be adding them into an annotation in the top-level class. See also scala.annotation.internal.SourceFile.

[japgolly]

Hey @nicolasstucki , would you be able to give me a bit of guidance? This was so easy to implement in Scala directly but I've been pulling my hair out trying to implement this as a compiler plugin. Three questions if that's ok?

1. I thought this would be a simple transformation but way too much happens in dotty.tools.dotc.typer.FrontEnd without giving me a chance. By the time a compiler plugin can start its transformation all of the inline methods have been expanded and typed (!). Instead I've made the runtime interface return a special hardcoded string and then in the compiler plugin I look around for it and replace it. It feels like such a fragile hack. Do you have any suggestions for how this could be done better?

2. I'm stuck trying to change the type of inline vals from the hardcoded magic string type, to the literal type of the value specified in plugin options. When I create the replacement ValDef, calling .tpt.tpe returns the new type I've inserted but I can't change .tpe.widenTermRefExpr. This is a problem because when we get to the PruneErasedDefs phase it just replaces my new type with the old one. You can see in the link above that After 1 is good but then After 2 returns the same value as Before :( How can I change .tpe.widenTermRefExpr?

3. I just realised: the way I'm seeing this work, it seems that the inline methods/vals that call my getEnv stuff, are being expanded before TASTY generation. Wouldn't this mean it wouldn't re-expand for downstream users with their config?

[nicolasstucki]

1. You will not be able to change the type of an inline val and that would be unsound and it technically impossible as all transparent inline definitions (including inline val) are inlined during typer. Maybe non-transparent inline definitions might be a better fit.

2. Maybe you can try to have a normal val which the plugging transforms into an inline val. This might allow non-transparent macros the new value. The idea would be to simply change the type of the val to a constant literal type.

- final val x: String = null
+ final val x: "abc" = "abc"

3. You will need to use non-transparent inline methods to have them expanded downstream.

[neko-kai]

Is the proposed -E flag to inlines essentially what -Xmacro-settings was to macros in Scala 2?
We use the latter extensively in Scala 2 to pass information about build environment to compiled code like this scalacOptions += s"-Xmacro-settings:is-ci=${insideCI.value}"

[Baccata]

I'm currently facing a problem that would be easily solved if I had access to the sourceroot or sourcepath compiler setting from the macro context (which is possible in Scala 2, via a call to compilerSettings value on a macro context).

Whilst it's not covered by this proposal, the problem is similar. Is there a concern for exposing the whole set of compiler settings via a metaprogramming API ?

[japgolly]

@neko-kai I didn't know that -Xmacro-settings was a thing! Yes that seems like exactly what I'm proposing!

@Baccata 👍

[neko-kai]

@Baccata I think you can retrieve sourcepath using

scala.quoted.quotes.reflect.Position.ofMacroExpansion.sourceFile.getJPath

This retrieves the absolute path to the current source file. I thihk the macro below solves a similar problem to yours - 7mind/sbtgen@e6f96a4#diff-1fa20673ba6904c5702fdceacd1fa15adf52b5e86dc66b4ccce8efebe3586b94R73

[pshirshov]

Pretty please. We actually need this.