dotc: Error: requirement failed: array index too large, maximum is 2^30 - 1

[pshirshov]

Sometimes I'm getting this stacktrace while running heavy TASTY macros:

dotc: Error: requirement failed: array index too large, maximum is 2^30 - 1
java.lang.IllegalArgumentException: requirement failed: array index too large, maximum is 2^30 - 1
	at scala.Predef$.require(Predef.scala:340)
	at dotty.tools.dotc.util.SparseIntArray.update(SparseIntArray.scala:32)
	at dotty.tools.dotc.core.tasty.TreeBuffer.registerTreeAddr(TreeBuffer.scala:34)
	at dotty.tools.dotc.core.tasty.TreePickler.pickleTree(TreePickler.scala:348)
	at dotty.tools.dotc.core.tasty.TreePickler.pickle$$anonfun$1(TreePickler.scala:736)
	at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at dotty.tools.dotc.core.tasty.TreePickler.pickle(TreePickler.scala:736)
	at dotty.tools.dotc.transform.Pickler.run$$anonfun$3$$anonfun$2(Pickler.scala:64)
	at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at dotty.tools.dotc.transform.Pickler.run$$anonfun$1(Pickler.scala:101)
	at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at dotty.tools.dotc.transform.Pickler.run(Pickler.scala:101)
	at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:296)
	at scala.collection.immutable.List.map(List.scala:246)
	at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:297)
	at dotty.tools.dotc.transform.Pickler.runOn(Pickler.scala:105)
	at dotty.tools.dotc.Run.runPhases$4$$anonfun$4(Run.scala:180)
	at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
	at dotty.tools.dotc.Run.runPhases$5(Run.scala:190)
	at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:198)
	at dotty.runtime.function.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
	at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:64)
	at dotty.tools.dotc.Run.compileUnits(Run.scala:205)
	at dotty.tools.dotc.Run.compileSources(Run.scala:142)
	at dotty.tools.dotc.Run.compile(Run.scala:124)
	at dotty.tools.dotc.Driver.doCompile(Driver.scala:38)
	at dotty.tools.dotc.Driver.process(Driver.scala:195)
	at dotty.tools.dotc.Main.process(Main.scala)
	at xsbt.CachedCompilerImpl.run(CachedCompilerImpl.java:69)
	at xsbt.CompilerInterface.run(CompilerInterface.java:41)
	at jdk.internal.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at sbt.internal.inc.AnalyzingCompiler.call(AnalyzingCompiler.scala:237)
	at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:111)
	at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:90)
	at org.jetbrains.jps.incremental.scala.local.IdeaIncrementalCompiler.compile(IdeaIncrementalCompiler.scala:42)
	at org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:43)
	at org.jetbrains.jps.incremental.scala.remote.Main$.compileLogic(Main.scala:145)
	at org.jetbrains.jps.incremental.scala.remote.Main$.$anonfun$handleCommand$1(Main.scala:131)
	at org.jetbrains.jps.incremental.scala.remote.Main$.decorated$1(Main.scala:121)
	at org.jetbrains.jps.incremental.scala.remote.Main$.handleCommand(Main.scala:128)
	at org.jetbrains.jps.incremental.scala.remote.Main$.serverLogic(Main.scala:105)
	at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:63)
	at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
	at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

I cannot provide a minimized example - the issue manifests occasionally, so it's really hard to make a minimized example.

It sometimes happens during izumi-reflect build. It may be a recent regression, I didn't see it before 0.27

[smarter]

Without more information there's nothing we can do. Are you actually generating more than 2^30 trees? If you look at -Xprint:pickler you should be able to see if your output is enormous or not.

[pshirshov]

Are you actually generating more than 2^30 trees?

I believe I don't. Moreover, I'm traversing trees, my output is, essentially, just a long string literal. When compilation succeeds the output is pretty normal and compact.

Without more information there's nothing we can do.

Sorry if this is of no interest for you. I had a hope that it may be easier for you to repeat the issue locally and debug it. At the moment I don't have more information and not sure if I can provide more - as I said before it's very hard to make a minimized example because the issue manifests occasionally.

[smarter]

A good first step would be to find a reliable way to reproduce the problem, even if it's not minimized.

[pshirshov]

And this is the primary problem. Though I just got into something interesting, seems like it is not related to macros. Last time it happened while building this: https://github.com/zio/izumi-reflect/tree/develop/izumi-reflect/izumi-reflect-thirdparty-boopickle-shaded

There are no macros at all.

[odersky]

I have a hypothesis what this could be. Did you use a long-running resident compiler? I noted that tree ids are not recycled between runs.

[neko-kai]

@odersky Yes, this tends to happen after a few compiles with either sbt or Intellij's build server.

[pshirshov]

@odersky : yes, usually it helps to restart the compile server, though sometimes it may happen right on the first run. Also seems like it fails while traversing the trees, not while producing new ones (I got to this conclusion with some probabilistic debug prints, but I may be wrong).

[pshirshov]

Thanks! Once next release is available I'll test it and report here.