GC overhead limit hit - Serializer should stream XML instead of keep everything in memory

[cridmann]

Hey guys,

I think this issue is related to a recently closed issue on sbt-scoverage: scoverage/sbt-scoverage#50

Basically, I'm using a multi-module play 2.3.x project. Scoverage works for some of the smaller modules, but not for larger ones. In particular, one that is failing has ~12k lines of code.

Right now I'm getting GC overhead limits hit, and it appears to be from the # of lines of code and the mechanism of serialization that is used. I have given sbt over 4GB of memory, and it still blows up. It shouldn't need this much.

Here is a small stack trace (if you need more, let me know):

[info] Instrumentation completed [26581 statements]
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:188)
at sbt.Execute.next$1(Execute.scala:83)
at sbt.Execute.processAll(Execute.scala:86)
at sbt.Execute.runKeep(Execute.scala:66)
at sbt.EvaluateTask$.liftedTree1$1(EvaluateTask.scala:342)
at sbt.EvaluateTask$.run$1(EvaluateTask.scala:341)
at sbt.EvaluateTask$.runTask(EvaluateTask.scala:361)
at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:64)
at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:62)
at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:293)
at sbt.Aggregation$.timedRun(Aggregation.scala:62)
at sbt.Aggregation$.runTasks(Aggregation.scala:71)
at sbt.Aggregation$$anonfun$applyTasks$1.apply(Aggregation.scala:32)
at sbt.Aggregation$$anonfun$applyTasks$1.apply(Aggregation.scala:31)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
at sbt.Aggregation$$anonfun$evaluatingParser$4$$anonfun$apply$5.apply(Aggregation.scala:153)
at sbt.Aggregation$$anonfun$evaluatingParser$4$$anonfun$apply$5.apply(Aggregation.scala:152)
at sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:244)
at sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:241)
at sbt.Command$.process(Command.scala:92)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
at sbt.State$$anon$1.process(State.scala:184)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.MainLoop$.next(MainLoop.scala:98)
at sbt.MainLoop$.run(MainLoop.scala:91)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:70)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:65)
at sbt.Using.apply(Using.scala:24)
at sbt.MainLoop$.runWithNewLog(MainLoop.scala:65)
at sbt.MainLoop$.runAndClearLast(MainLoop.scala:48)
at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:32)
at sbt.MainLoop$.runLogged(MainLoop.scala:24)
at sbt.StandardMain$.runManaged(Main.scala:53)
at sbt.xMain.run(Main.scala:28)
at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:57)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:77)
at xsbt.boot.Launch$.run(Launch.scala:57)
at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:45)
at xsbt.boot.Launch$.launch(Launch.scala:65)
at xsbt.boot.Launch$.apply(Launch.scala:16)
at xsbt.boot.Boot$.runImpl(Boot.scala:32)
at xsbt.boot.Boot$.main(Boot.scala:21)
at xsbt.boot.Boot.main(Boot.scala)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at scala.xml.Elem$.unapplySeq(Elem.scala:38)
at scala.xml.Utility$.trimProper(Utility.scala:58)
at scala.xml.Utility$$anonfun$4.apply(Utility.scala:59)
at scala.xml.Utility$$anonfun$4.apply(Utility.scala:59)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:252)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at scala.xml.Utility$.trimProper(Utility.scala:59)
at scala.xml.Utility$$anonfun$3.apply(Utility.scala:49)
at scala.xml.Utility$$anonfun$3.apply(Utility.scala:49)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:252)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at scala.xml.Utility$.trim(Utility.scala:49)
at scoverage.Serializer$.serialize(Serializer.scala:72)
at scoverage.Serializer$.serialize(Serializer.scala:14)
at scoverage.ScoverageInstrumentationComponent$$anon$1.run(plugin.scala:96)
at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1338)
at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1325)
at scala.tools.nsc.Global$Run.compileSources(Global.scala:1320)
at scala.tools.nsc.Global$Run.compile(Global.scala:1418)
at xsbt.CachedCompiler0.run(CompilerInterface.scala:116)
at xsbt.CachedCompiler0.run(CompilerInterface.scala:95)
at xsbt.CompilerInterface.run(CompilerInterface.scala:26)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded

Thanks for the great work so far...let me know if you have any questions

[kryptt]

I am also afected by this issue! great to see 38f7c78 !!

[cridmann]

If you are using sbt-scoverage, I think this patch was included in 1.0.4 Once I upgraded to that version, the issue was resolved