Skip to content

scaladoc hangs #17577

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
reid-spencer opened this issue May 24, 2023 · 2 comments
Closed

scaladoc hangs #17577

reid-spencer opened this issue May 24, 2023 · 2 comments
Assignees
@Florian3k
Labels
area:doctool itype:bug

Comments

@reid-spencer
Copy link

reid-spencer commented May 24, 2023 •
edited
Loading

Compiler version

3.2.2
3.3.0-RC5
3.3.0-RC6
3.3.0
3.3.1-RC1

Minimized code

Not available yet. Extracting from 100s of classes is time-consuming.

Follow my analysis here:
https://contributors.scala-lang.org/t/scala3doc-spin-loop/6202/7

Output

None other than:

[info] Main Scala API documentation to /Users/reid/Code/reactific/riddl/language/target/scala-3.3.0/api...

Expectation

The compiler completes the documentation generation instead of hanging forever.
@reid-spencer reid-spencer added itype:bug stat:needs triage Every issue needs to have an "area" and "itype" label labels May 24, 2023
@reid-spencer
Copy link
Author

To reproduce:

That process will never end and a java process will run at > 100% CPU utilization forever (many hours tested).

@szymon-rd szymon-rd added area:doctool and removed stat:needs triage Every issue needs to have an "area" and "itype" label labels May 29, 2023
@reid-spencer reid-spencer mentioned this issue May 30, 2023
Open
@reardonj
Copy link
Contributor

I took a bit of a look at this and have a minimization. I started by getting a thread dump while generating docs with from the affected code. Scalac appears to be getting stuck in the InheritanceInformationTransformer code of scaladoc generation, performing concatenation and distinct calls on Seq. I haven't delved into the scalac code yet, but it smells like an exponential blowup when there is a large amount of inheritance. The affected code includes a deep inheritance heirarchy with a series of traits, each defining inner traits, classes, and objects. Changing the design to use self types instead allows the documentation generation to complete.

This minimization only takes several minutes instead of hanging indefinitely, but hopefully gets the idea across compactly: https://github.com/reardonj/minimize-dotty-bug-17577

"pool-77-thread-11" #610 prio=5 os_prio=31 cpu=80053.92ms elapsed=83.12s tid=0x00007fa40db38800 nid=0xaf67 runnable  [0x0000700013958000]
   java.lang.Thread.State: RUNNABLE
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.util.hashing.MurmurHash3.productHash(MurmurHash3.scala:76)
        at scala.util.hashing.MurmurHash3$.productHash(MurmurHash3.scala:343)
        at scala.runtime.ScalaRunTime$._hashCode(ScalaRunTime.scala:158)
        at dotty.tools.scaladoc.Kind$Trait.hashCode(api.scala:60)
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.util.hashing.MurmurHash3.productHash(MurmurHash3.scala:76)
        at scala.util.hashing.MurmurHash3$.productHash(MurmurHash3.scala:343)
        at scala.runtime.ScalaRunTime$._hashCode(ScalaRunTime.scala:158)
        at dotty.tools.scaladoc.LinkToType.hashCode(api.scala:141)
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.util.hashing.MurmurHash3.productHash(MurmurHash3.scala:76)
        at scala.util.hashing.MurmurHash3$.productHash(MurmurHash3.scala:343)
        at scala.Tuple2.hashCode(Tuple2.scala:24)
        at scala.runtime.Statics.anyHash(Statics.java:127)
        at scala.collection.mutable.HashSet.add(HashSet.scala:69)
        at scala.collection.immutable.StrictOptimizedSeqOps.distinctBy(StrictOptimizedSeqOps.scala:35)
        at scala.collection.immutable.StrictOptimizedSeqOps.distinctBy$(StrictOptimizedSeqOps.scala:26)
        at scala.collection.immutable.List.distinctBy(List.scala:79)
        at scala.collection.SeqOps.distinct(Seq.scala:206)
        at scala.collection.SeqOps.distinct$(Seq.scala:206)
        at scala.collection.AbstractSeq.distinct(Seq.scala:1188)
        at dotty.tools.scaladoc.HierarchyGraph.$plus(api.scala:146)
        at dotty.tools.scaladoc.HierarchyGraph.$plus$plus$$anonfun$1(api.scala:148)
        at dotty.tools.scaladoc.HierarchyGraph$$Lambda$6734/0x0000000801cf2040.apply(Unknown Source)
        at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
        at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
        at scala.collection.immutable.List.foldLeft(List.scala:79)
        at dotty.tools.scaladoc.HierarchyGraph.$plus$plus(api.scala:149)
        at dotty.tools.scaladoc.api$package$.withNewGraphEdges(api.scala:220)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer.apply$$anonfun$1(InheritanceInformationTransformer.scala:10)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer$$Lambda$6910/0x0000000801dca040.apply(Unknown Source)
        at dotty.tools.scaladoc.api$package$.updateRecusivly(api.scala:205)
        at dotty.tools.scaladoc.api$package$.$anonfun$1(api.scala:204)
        at dotty.tools.scaladoc.api$package$$$Lambda$6913/0x0000000801dcc840.apply(Unknown Source)
        at scala.collection.immutable.List.map(List.scala:250)
        at scala.collection.immutable.List.map(List.scala:79)
        at dotty.tools.scaladoc.api$package$.updateRecusivly(api.scala:204)
        at dotty.tools.scaladoc.api$package$.updateMembers$$anonfun$1$$anonfun$1(api.scala:240)
        at dotty.tools.scaladoc.api$package$$$Lambda$6912/0x0000000801dcb840.apply(Unknown Source)
        at scala.collection.immutable.List.map(List.scala:250)
        at scala.collection.immutable.List.map(List.scala:79)
        at dotty.tools.scaladoc.api$package$.updateMembers$$anonfun$1(api.scala:240)
        at dotty.tools.scaladoc.api$package$$$Lambda$6911/0x0000000801dcb040.apply(Unknown Source)
        at dotty.tools.scaladoc.api$package$.updatePackages(api.scala:236)
        at dotty.tools.scaladoc.api$package$.updateMembers(api.scala:240)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer.apply(InheritanceInformationTransformer.scala:10)
        at dotty.tools.scaladoc.transformers.InheritanceInformationTransformer.apply(InheritanceInformationTransformer.scala:5)
        at dotty.tools.scaladoc.ScalaModuleProvider$.mkModule$$anonfun$1(ScalaModuleProvider.scala:51)
        at dotty.tools.scaladoc.ScalaModuleProvider$$$Lambda$6894/0x0000000801db7840.apply(Unknown Source)
        at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
        at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
        at scala.collection.immutable.List.foldLeft(List.scala:79)
        at dotty.tools.scaladoc.ScalaModuleProvider$.mkModule(ScalaModuleProvider.scala:51)
        at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:241)
        at dotty.tools.scaladoc.Scaladoc$.run$$anonfun$1(Scaladoc.scala:73)
        at dotty.tools.scaladoc.Scaladoc$$$Lambda$6622/0x0000000801c67040.applyVoid(Unknown Source)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.Option.map(Option.scala:242)
        at dotty.tools.scaladoc.Scaladoc$.run(Scaladoc.scala:77)
        at dotty.tools.dottydoc.Main$.process(Main.scala:25)
        at dotty.tools.dottydoc.Main.process(Main.scala)
        at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
        at xsbt.DottydocRunner.run(DottydocRunner.java:61)
        at xsbt.ScaladocInterface.run(ScaladocInterface.java:11)
        at jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
        at sbt.internal.inc.AnalyzingCompiler.invoke(AnalyzingCompiler.scala:329)
        at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:175)
        at sbt.internal.inc.AnalyzingCompiler.doc(AnalyzingCompiler.scala:133)
        at sbt.Doc$.$anonfun$scaladoc$1(Doc.scala:52)
        at sbt.Doc$.$anonfun$scaladoc$1$adapted(Doc.scala:40)
        at sbt.Doc$$$Lambda$6575/0x0000000801c35c40.apply(Unknown Source)
        at sbt.RawCompileLike$.$anonfun$prepare$1(RawCompileLike.scala:79)
        at sbt.RawCompileLike$.$anonfun$prepare$1$adapted(RawCompileLike.scala:72)
        at sbt.RawCompileLike$$$Lambda$6576/0x0000000801c36040.apply(Unknown Source)
        at sbt.RawCompileLike$.$anonfun$cached$4(RawCompileLike.scala:63)
        at sbt.RawCompileLike$.$anonfun$cached$4$adapted(RawCompileLike.scala:61)
        at sbt.RawCompileLike$$$Lambda$6584/0x0000000801c3d840.apply(Unknown Source)
        at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:219)
        at sbt.util.Tracked$$$Lambda$2940/0x0000000800db9840.apply(Unknown Source)
        at sbt.RawCompileLike$.$anonfun$cached$1(RawCompileLike.scala:68)
        at sbt.RawCompileLike$.$anonfun$cached$1$adapted(RawCompileLike.scala:52)
        at sbt.RawCompileLike$$$Lambda$6577/0x0000000801c36440.apply(Unknown Source)
        at sbt.Defaults$.$anonfun$docTaskSettings$4(Defaults.scala:2159)
        at sbt.Defaults$$$Lambda$811/0x0000000800640040.apply(Unknown Source)
        at scala.Function1.$anonfun$compose$1(Function1.scala:49)
        at scala.Function1$$Lambda$339/0x000000080040b040.apply(Unknown Source)
        at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
        at sbt.internal.util.$tilde$greater$$Lambda$2773/0x0000000800d53840.apply(Unknown Source)
        at sbt.std.Transform$$anon$4.work(Transform.scala:68)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
        at sbt.Execute$$Lambda$2799/0x0000000800d60c40.apply(Unknown Source)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
        at sbt.Execute.work(Execute.scala:291)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
        at sbt.Execute$$Lambda$2783/0x0000000800d59440.apply(Unknown Source)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
        at sbt.ConcurrentRestrictions$$anon$4$$Lambda$2796/0x0000000800d5f840.apply(Unknown Source)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
        at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
        at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:515)
        at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run([email protected]/Thread.java:829)

reardonj added a commit to reardonj/dotty that referenced this issue Jun 11, 2023
@reardonj
Closes scala#17577: scaladoc hangs with deep inheritance
866d2f9 
This commit has a pair of fixes that bring the scaladoc build time for the repository mentioned in scala#17577 to a minute instead of hanging.

- Reimplement HierarchyGraph's `+` method in terms of `++` Previously, `++` was implemented in terms of `+`, but `+` called `distinct` resulting in poor performance. Now `distinct` is only called once for a `++` call.
- Use Set instead of Seq for `InheritanceInformationTransformer.getEdges`. Previously, `subtypes` could contain duplicates, causing excess calls to `getEdges`

Use Set in InheritanceInformationTransformer
reardonj added a commit to reardonj/dotty that referenced this issue Jun 11, 2023
@reardonj
Closes scala#17577: scaladoc hangs with deep inheritance
cf0592b 
This commit has a pair of fixes that bring the scaladoc build time for the repository mentioned in scala#17577 to a minute instead of hanging.

- Reimplement HierarchyGraph's `+` method in terms of `++` Previously, `++` was implemented in terms of `+`, but `+` called `distinct` resulting in poor performance. Now `distinct` is only called once for a `++` call.
- Use Set instead of Seq for `InheritanceInformationTransformer.getEdges`. Previously, `subtypes` could contain duplicates, causing excess calls to `getEdges`
reardonj added a commit to reardonj/dotty that referenced this issue Jun 11, 2023
@reardonj
Fix scala#17577: scaladoc hangs with deep inheritance
b668562 
This commit has a pair of fixes that bring the scaladoc build time for the repository mentioned in scala#17577 to a minute instead of hanging.

- Reimplement HierarchyGraph's `+` method in terms of `++` Previously, `++` was implemented in terms of `+`, but `+` called `distinct` resulting in poor performance. Now `distinct` is only called once for a `++` call.
- Use Set instead of Seq for `InheritanceInformationTransformer.getEdges`. Previously, `subtypes` could contain duplicates, causing excess calls to `getEdges`
@reardonj reardonj mentioned this issue Jun 11, 2023
Merged
Florian3k added a commit that referenced this issue Jun 16, 2023
@Florian3k
Fix #17577: scaladoc hangs with deep inheritance (#17954)
e7fa5dc 
This commit has a pair of fixes that bring the scaladoc build time for
the repository mentioned in #17577 to a minute instead of hanging.

- Reimplement HierarchyGraph's `+` method in terms of `++` Previously,
`++` was implemented in terms of `+`, but `+` called `distinct`
resulting in poor performance. Now `distinct` is only called once for a
`++` call.
- Get the distinct `LinkToType` values in
`InheritanceInformationTransformer.apply`. Previously, `subtypes` could
contain duplicates, causing redundant calls to `getEdges`
Kordyjan pushed a commit that referenced this issue Nov 17, 2023
@reardonj @Kordyjan
Fix #17577: scaladoc hangs with deep inheritance
d3266d8 
This commit has a pair of fixes that bring the scaladoc build time for the repository mentioned in #17577 to a minute instead of hanging.

- Reimplement HierarchyGraph's `+` method in terms of `++` Previously, `++` was implemented in terms of `+`, but `+` called `distinct` resulting in poor performance. Now `distinct` is only called once for a `++` call.
- Get the distinct `LinkToType` values in `InheritanceInformationTransformer.apply`. Previously, `subtypes` could contain duplicates, causing redundant  calls to `getEdges`

[Cherry-picked 9859e89]
Kordyjan added a commit that referenced this issue Nov 21, 2023
@Kordyjan
Backport "Fix #17577: scaladoc hangs with deep inheritance" to LTS (#…
6bfe99b 
…18991)

Backports #17954 to the LTS branch.

PR submitted by the release tooling.
[skip ci]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Florian3k Florian3k

Labels
area:doctool itype:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@reardonj @reid-spencer @szymon-rd @Florian3k