Runtime staging compiler crash when creating an object using "new" (scala 3.3.1) #19170

christophkoch · 2023-12-01T22:33:40Z

Compiler version

3.3.1

Minimized code

import scala.quoted.*
given staging.Compiler = staging.Compiler.make(getClass.getClassLoader)

class A(i: Int)

def f(i: Expr[Int])(using Quotes): Expr[A] = { '{ new A($i) } }

val g: Int => A = staging.run { '{ (i: Int) => ${ f('{i}) } } }

Output

scala> g
dotty.tools.dotc.reporting.UnhandledError: undefined: new A # -1: TermRef(TypeRef(ThisType(TypeRef(ThisType(TypeRef(NoPrefix,module class )),module class )),A),) at quotedFrontend
at dotty.tools.dotc.reporting.ThrowingReporter.doReport(ThrowingReporter.scala:14)
at dotty.tools.dotc.reporting.Reporter.issueUnconfigured(Reporter.scala:158)
at dotty.tools.dotc.reporting.Reporter.go$1(Reporter.scala:181)
at dotty.tools.dotc.reporting.Reporter.issueIfNotSuppressed(Reporter.scala:200)
at dotty.tools.dotc.reporting.Reporter.report(Reporter.scala:203)
at dotty.tools.dotc.report$.error(report.scala:68)
at dotty.tools.dotc.typer.ErrorReporting$.errorType(ErrorReporting.scala:31)
at dotty.tools.dotc.typer.TypeAssigner.assignType(TypeAssigner.scala:298)
at dotty.tools.dotc.typer.TypeAssigner.assignType$(TypeAssigner.scala:16)
at dotty.tools.dotc.typer.Typer.assignType(Typer.scala:116)
at dotty.tools.dotc.ast.tpd$.Apply(tpd.scala:49)
at dotty.tools.dotc.core.tasty.TreeUnpickler$TreeReader.readLengthTree$1(TreeUnpickler.scala:1304)
at dotty.tools.dotc.core.tasty.TreeUnpickler$TreeReader.readTree(TreeUnpickler.scala:1468)
at dotty.tools.dotc.core.tasty.TreeUnpickler$TreeReader.readLengthTree$1(TreeUnpickler.scala:1333)
at dotty.tools.dotc.core.tasty.TreeUnpickler$TreeReader.readTree(TreeUnpickler.scala:1468)
at dotty.tools.dotc.core.tasty.TreeUnpickler.unpickle(TreeUnpickler.scala:117)
at dotty.tools.dotc.core.tasty.DottyUnpickler.computeRootTrees(DottyUnpickler.scala:62)
at dotty.tools.dotc.ast.tpd$TreeProvider.rootTrees(tpd.scala:1299)
at dotty.tools.dotc.ast.tpd$TreeProvider.rootTrees$(tpd.scala:1288)
at dotty.tools.dotc.core.tasty.DottyUnpickler.rootTrees(DottyUnpickler.scala:44)
at dotty.tools.dotc.ast.tpd$TreeProvider.tree(tpd.scala:1303)
at dotty.tools.dotc.ast.tpd$TreeProvider.tree$(tpd.scala:1288)
at dotty.tools.dotc.core.tasty.DottyUnpickler.tree(DottyUnpickler.scala:44)
at dotty.tools.dotc.quoted.PickledQuotes$.unpickle(PickledQuotes.scala:274)
at dotty.tools.dotc.quoted.PickledQuotes$.unpickleTerm(PickledQuotes.scala:84)
at scala.quoted.runtime.impl.QuotesImpl.unpickleExprV2(QuotesImpl.scala:3143)
at Meta$package$.f(Meta.scala:7)
at Meta$package$.$init$$$anonfun$1$$anonfun$1$$anonfun$1(Meta.scala:9)
at Meta$package$.$init$$$anonfun$1$$anonfun$1(Meta.scala:9)
at Meta$package$.$init$$$anonfun$1$$anonfun$adapted$1(Meta.scala:9)
at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:110)
at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1538)
at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:135)
at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transformBlock(Trees.scala:1603)
at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1518)
at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:135)
at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1570)
at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:135)
at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform$$anonfun$1(Trees.scala:1605)
at scala.collection.immutable.List.mapConserve(List.scala:472)
at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1605)
at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transformStats(Trees.scala:1601)
at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transformBlock(Trees.scala:1603)
at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1518)
at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:135)
at dotty.tools.dotc.quoted.PickledQuotes$.spliceTerms(PickledQuotes.scala:152)
at dotty.tools.dotc.quoted.PickledQuotes$.unpickleTerm(PickledQuotes.scala:88)
at scala.quoted.runtime.impl.QuotesImpl.unpickleExprV2(QuotesImpl.scala:3143)
at Meta$package$.$init$$$anonfun$1(Meta.scala:9)
at scala.quoted.staging.QuoteCompiler$QuotedFrontend.runOn$$anonfun$1(QuoteCompiler.scala:83)
at scala.collection.immutable.List.flatMap(List.scala:293)
at scala.quoted.staging.QuoteCompiler$QuotedFrontend.runOn(QuoteCompiler.scala:98)
at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:246)
at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1321)
at dotty.tools.dotc.Run.runPhases$1(Run.scala:262)
at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:270)
at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:279)
at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:67)
at dotty.tools.dotc.Run.compileUnits(Run.scala:279)
at dotty.tools.dotc.Run.compileUnits(Run.scala:200)
at scala.quoted.staging.QuoteCompiler$ExprRun.compileExpr(QuoteCompiler.scala:118)
at scala.quoted.staging.QuoteDriver.run(QuoteDriver.scala:43)
at scala.quoted.staging.Compiler$$anon$1.run(Compiler.scala:38)
at scala.quoted.staging.package$.run(staging.scala:19)
at Meta$package$.(Meta.scala:9)
... 66 elided

Expectation

...for this not to happen.

hamzaremmal · 2023-12-01T23:22:57Z

Compiling the file below (i19170.scala) will emit a valid warning

import scala.quoted.*
given staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
class A(i: Int)
def f(i: Expr[Int])(using Quotes): Expr[A] = { '{ new A($i) } }
val g: Int => A = staging.run { '{ (i: Int) => ${ f('{i}) } } }
@main def run = g

Full stack trace when running scala i19170.scala:

-- [E181] Potential Issue Warning: /Users/hamzaremmal/Desktop/LAMP/dotty/i19170.scala:3:47
3 |given staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
  |                                               ^^^^^^^^
  |                         Suspicious top-level unqualified call to getClass
  |
  | longer explanation available when compiling with `-explain`
1 warning found
java.lang.ExceptionInInitializerError
	at run.main(i19170.scala:11)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:33)
	at dotty.tools.scripting.Main$.process(Main.scala:45)
	at dotty.tools.MainGenericRunner$.run$1(MainGenericRunner.scala:250)
	at dotty.tools.MainGenericRunner$.process(MainGenericRunner.scala:270)
	at dotty.tools.MainGenericRunner$.main(MainGenericRunner.scala:281)
	at dotty.tools.MainGenericRunner.main(MainGenericRunner.scala)
Caused by: dotty.tools.dotc.reporting.UnhandledError: undefined: new A # -1: TermRef(TypeRef(ThisType(TypeRef(ThisType(TypeRef(NoPrefix,module class <root>)),module class <empty>)),A),<init>) at quotedFrontend
	at dotty.tools.dotc.reporting.ThrowingReporter.doReport(ThrowingReporter.scala:14)
	at dotty.tools.dotc.reporting.Reporter.issueUnconfigured(Reporter.scala:158)
	at dotty.tools.dotc.reporting.Reporter.go$1(Reporter.scala:181)
	at dotty.tools.dotc.reporting.Reporter.issueIfNotSuppressed(Reporter.scala:200)
	at dotty.tools.dotc.reporting.Reporter.report(Reporter.scala:203)
	at dotty.tools.dotc.report$.error(report.scala:68)
	at dotty.tools.dotc.typer.ErrorReporting$.errorType(ErrorReporting.scala:31)
	at dotty.tools.dotc.typer.TypeAssigner.assignType(TypeAssigner.scala:298)
	at dotty.tools.dotc.typer.TypeAssigner.assignType$(TypeAssigner.scala:16)
	at dotty.tools.dotc.typer.Typer.assignType(Typer.scala:116)
	at dotty.tools.dotc.ast.tpd$.Apply(tpd.scala:49)
	at dotty.tools.dotc.core.tasty.TreeUnpickler$TreeReader.readLengthTree$1(TreeUnpickler.scala:1304)
	at dotty.tools.dotc.core.tasty.TreeUnpickler$TreeReader.readTree(TreeUnpickler.scala:1468)
	at dotty.tools.dotc.core.tasty.TreeUnpickler$TreeReader.readLengthTree$1(TreeUnpickler.scala:1333)
	at dotty.tools.dotc.core.tasty.TreeUnpickler$TreeReader.readTree(TreeUnpickler.scala:1468)
	at dotty.tools.dotc.core.tasty.TreeUnpickler.unpickle(TreeUnpickler.scala:117)
	at dotty.tools.dotc.core.tasty.DottyUnpickler.computeRootTrees(DottyUnpickler.scala:62)
	at dotty.tools.dotc.ast.tpd$TreeProvider.rootTrees(tpd.scala:1299)
	at dotty.tools.dotc.ast.tpd$TreeProvider.rootTrees$(tpd.scala:1288)
	at dotty.tools.dotc.core.tasty.DottyUnpickler.rootTrees(DottyUnpickler.scala:44)
	at dotty.tools.dotc.ast.tpd$TreeProvider.tree(tpd.scala:1303)
	at dotty.tools.dotc.ast.tpd$TreeProvider.tree$(tpd.scala:1288)
	at dotty.tools.dotc.core.tasty.DottyUnpickler.tree(DottyUnpickler.scala:44)
	at dotty.tools.dotc.quoted.PickledQuotes$.unpickle(PickledQuotes.scala:274)
	at dotty.tools.dotc.quoted.PickledQuotes$.unpickleTerm(PickledQuotes.scala:84)
	at scala.quoted.runtime.impl.QuotesImpl.unpickleExprV2(QuotesImpl.scala:3143)
	at i19170$package$.f(i19170.scala:7)
	at i19170$package$.$init$$$anonfun$1$$anonfun$1$$anonfun$1(i19170.scala:9)
	at i19170$package$.$init$$$anonfun$1$$anonfun$1(i19170.scala:9)
	at i19170$package$.$init$$$anonfun$1$$anonfun$adapted$1(i19170.scala:9)
	at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:110)
	at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1538)
	at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:135)
	at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transformBlock(Trees.scala:1603)
	at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1518)
	at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:135)
	at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1570)
	at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:135)
	at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform$$anonfun$1(Trees.scala:1605)
	at scala.collection.immutable.List.mapConserve(List.scala:472)
	at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1605)
	at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transformStats(Trees.scala:1601)
	at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transformBlock(Trees.scala:1603)
	at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1518)
	at dotty.tools.dotc.quoted.PickledQuotes$$anon$1.transform(PickledQuotes.scala:135)
	at dotty.tools.dotc.quoted.PickledQuotes$.spliceTerms(PickledQuotes.scala:152)
	at dotty.tools.dotc.quoted.PickledQuotes$.unpickleTerm(PickledQuotes.scala:88)
	at scala.quoted.runtime.impl.QuotesImpl.unpickleExprV2(QuotesImpl.scala:3143)
	at i19170$package$.$init$$$anonfun$1(i19170.scala:9)
	at scala.quoted.staging.QuoteCompiler$QuotedFrontend.runOn$$anonfun$1(QuoteCompiler.scala:83)
	at scala.collection.immutable.List.flatMap(List.scala:293)
	at scala.quoted.staging.QuoteCompiler$QuotedFrontend.runOn(QuoteCompiler.scala:98)
	at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:246)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1321)
	at dotty.tools.dotc.Run.runPhases$1(Run.scala:262)
	at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:270)
	at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:279)
	at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:67)
	at dotty.tools.dotc.Run.compileUnits(Run.scala:279)
	at dotty.tools.dotc.Run.compileUnits(Run.scala:200)
	at scala.quoted.staging.QuoteCompiler$ExprRun.compileExpr(QuoteCompiler.scala:118)
	at scala.quoted.staging.QuoteDriver.run(QuoteDriver.scala:43)
	at scala.quoted.staging.Compiler$$anon$1.run(Compiler.scala:38)
	at scala.quoted.staging.package$.run(staging.scala:19)
	at i19170$package$.<clinit>(i19170.scala:9)
	... 11 more

hamzaremmal · 2023-12-01T23:35:01Z

Running it on the main branch (d96e9e4) gives a different output :

-- [E006] Not Found Error: i19170.scala:3:6 ------------------------------------
3 |given staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
  |      ^^^^^^^
  |      Not found: staging - did you mean String?
  |
  | longer explanation available when compiling with `-explain`
-- [E006] Not Found Error: i19170.scala:3:25 -----------------------------------
3 |given staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
  |                         ^^^^^^^
  |                         Not found: staging - did you mean String?
  |
  | longer explanation available when compiling with `-explain`
-- [E006] Not Found Error: i19170.scala:9:18 -----------------------------------
9 |val g: Int => A = staging.run { '{ (i: Int) => ${ f('{i}) } } }
  |                  ^^^^^^^
  |                  Not found: staging - did you mean String?
  |
  | longer explanation available when compiling with `-explain`
3 errors found

nicolasstucki · 2023-12-02T07:33:03Z

@hamzaremmal you are missing the dependency on the staging jar. You should use tests/pos-staging to write tests with that dependency .

You can also use the -with-compiler flag.

christophkoch · 2023-12-02T12:47:16Z

Sorry, my code wasn't minimal. This smaller snippet still causes the crash:

import scala.quoted.*
given staging.Compiler = staging.Compiler.make(getClass.getClassLoader)

class A
val g: A = staging.run { '{ new A } }

Also, I would be curious to learn from the experts how to get rid of the warning about the "given staging.Compiler = ..." line, which Hamza also reported. This line is boilerplate I copied in from the latest online documentation, which may be outdated.

hamzaremmal · 2023-12-02T13:27:16Z

Also, I would be curious to learn from the experts how to get rid of the warning about the "given staging.Compiler = ..." line, which Hamza also reported.

The warning comes from the fact that you're calling a method from the top level, in this case getClass, and that this function call might end up calling the wrong method. That's why the message of the warning contains the following "Suspicious top-level unqualified call to getClass". The best way to get rid of this warning is to qualify the call to getClass, eg. Predef.getClass

@nicolasstucki Do you think this can be solved during a Spree ?

christophkoch · 2023-12-02T13:33:10Z

Thanks a lot, Hamza! It would be good to fix this in the documentation, e.g. at
https://docs.scala-lang.org/scala3/reference/metaprogramming/staging.html

nicolasstucki · 2024-01-11T17:16:59Z

The warning comes from the fact that you're calling a method from the top level, in this case getClass, and that this function call might end up calling the wrong method. That's why the message of the warning contains the following "Suspicious top-level unqualified call to getClass". The best way to get rid of this warning is to qualify the call to getClass, eg. Predef.getClass

This points to the actual issue with this code, but it does not make the solution evident. In the past, when we called getClass in a top-level method we used to call this.getClass where this was on the synthetic package object. Now we get a Predef.getClass.

What we need to pass to staging.Compiler.make is the classloader that loaded the application. Some tools (scala, SBT, or others) load the standard library before the application is loaded in a parent classloader. If we use this classloader, the runtime staging compiler will not know how to load the classes of the application.

This seems to be the simplest workaround for top-level definitions

given staging.Compiler =
  object Dummy
  staging.Compiler.make(Dummy.getClass.getClassLoader)

We should update the documentation using this version, unless we find a simpler safe incantation for this classloader.

christophkoch · 2024-01-12T07:41:49Z

Thanks a lot Nicolas. If I do this in the Repl, I get

dotty.tools.dotc.MissingCoreLibraryException: Could not find package scala from compiler core libraries.
Make sure the compiler core libraries are on the classpath.

Is there an easy way to fix this?

This addresses part of scala#19211, scala#19170, and scala#19176

nicolasstucki · 2024-01-12T08:43:41Z

How are you executing the repl? In an SBT project or from the scala command?

dotty.tools.dotc.MissingCoreLibraryException: Could not find package scala from compiler core libraries.
Make sure the compiler core libraries are on the classpath.

A patch for this could be to add explicitly a dependency on the compiler.

Unfortunately, there seem to be other issues related to the classloader in the REPL.

nicolasstucki · 2024-01-12T08:46:07Z

I suspect that the issue in the REPL is located in https://github.com/lampepfl/dotty/blob/main/compiler/src/dotty/tools/dotc/util/ClasspathFromClassloader.scala

Fixes #19211 This addresses part of #19170, and #19176

This addresses part of #19211, #19170, and #19176 [Cherry-picked 693a80a]

christophkoch added itype:bug stat:needs triage Every issue needs to have an "area" and "itype" label labels Dec 1, 2023

hamzaremmal added area:metaprogramming:quotes Issues related to quotes and splices and removed stat:needs triage Every issue needs to have an "area" and "itype" label labels Dec 1, 2023

Bersier mentioned this issue Dec 6, 2023

Staging code example raises compiler warning #19211

Closed

nicolasstucki mentioned this issue Jan 12, 2024

Runtime staging cannot handle compiling functions with a second non-primitive argument #19176

Closed

nicolasstucki added a commit to dotty-staging/dotty that referenced this issue Jan 12, 2024

Update staging.Compiler.make documentation

693a80a

This addresses part of scala#19211, scala#19170, and scala#19176

nicolasstucki mentioned this issue Jan 12, 2024

Update staging.Compiler.make documentation #19428

Merged

nicolasstucki added a commit that referenced this issue Jan 18, 2024

Update staging.Compiler.make documentation (#19428)

34628b3

Fixes #19211 This addresses part of #19170, and #19176

Gedochao assigned jchyb Apr 4, 2024

jchyb mentioned this issue Apr 9, 2024

Improve reporting in staging about the possible use of an incorrect class loader #20137

Merged

jchyb closed this as completed in #20137 Apr 24, 2024

Kordyjan added this to the 3.5.0 milestone May 10, 2024

WojciechMazur pushed a commit that referenced this issue Jun 27, 2024

Update staging.Compiler.make documentation

3005598

This addresses part of #19211, #19170, and #19176 [Cherry-picked 693a80a]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Runtime staging compiler crash when creating an object using "new" (scala 3.3.1) #19170

Runtime staging compiler crash when creating an object using "new" (scala 3.3.1) #19170

christophkoch commented Dec 1, 2023

hamzaremmal commented Dec 1, 2023 •

edited

Loading

Uh oh!

hamzaremmal commented Dec 1, 2023

Uh oh!

nicolasstucki commented Dec 2, 2023

Uh oh!

christophkoch commented Dec 2, 2023

Uh oh!

hamzaremmal commented Dec 2, 2023

Uh oh!

christophkoch commented Dec 2, 2023

Uh oh!

nicolasstucki commented Jan 11, 2024

Uh oh!

christophkoch commented Jan 12, 2024

Uh oh!

nicolasstucki commented Jan 12, 2024

Uh oh!

nicolasstucki commented Jan 12, 2024

Uh oh!

Runtime staging compiler crash when creating an object using "new" (scala 3.3.1) #19170

Runtime staging compiler crash when creating an object using "new" (scala 3.3.1) #19170

Comments

christophkoch commented Dec 1, 2023

Compiler version

Minimized code

Output

Expectation

hamzaremmal commented Dec 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hamzaremmal commented Dec 1, 2023

Uh oh!

nicolasstucki commented Dec 2, 2023

Uh oh!

christophkoch commented Dec 2, 2023

Uh oh!

hamzaremmal commented Dec 2, 2023

Uh oh!

christophkoch commented Dec 2, 2023

Uh oh!

nicolasstucki commented Jan 11, 2024

Uh oh!

christophkoch commented Jan 12, 2024

Uh oh!

nicolasstucki commented Jan 12, 2024

Uh oh!

nicolasstucki commented Jan 12, 2024

Uh oh!

hamzaremmal commented Dec 1, 2023 •

edited

Loading