dotty compilation stackoverflow for huge number of cases classes

[liufengyun]

When I try to compile following file, dotty generates stackoverflow:

https://github.com/scala/scala/blob/2.12.x/test/files/neg/patmatexhaust-huge.scala

Stackoverflow happens after patmat and before erasure, one of following transform is dubious:

List(new VCInlineMethods,     // Inlines calls to value class methods
       new SeqLiterals,         // Express vararg arguments as arrays
       new InterceptedMethods,  // Special handling of `==`, `|=`, `getClass` methods
       new Getters,             // Replace non-private vals and vars with getter defs (fields are added later)
       new ElimByName,          // Expand by-name parameters and arguments
       new AugmentScala2Traits, // Expand traits defined in Scala 2.11 to simulate old-style rewritings
       new ResolveSuper,        // Implement super accessors and add forwarders to trait methods
       new ArrayConstructors),  // Intercept creation of (non-generic) arrays and intrinsify.

[odersky]

Suggestion: try to split the list to find out which mini=phase causes the stack overflow. E.g.

List(new VCInlineMethods,     // Inlines calls to value class methods
   new SeqLiterals,         // Express vararg arguments as arrays
   new InterceptedMethods,  // Special handling of `==`, `|=`, `getClass` methods
   new Getters,             // Replace non-private vals and vars with getter defs (fields are added later)
   new ElimByName),   // Expand by-name parameters and arguments
List(
   new AugmentScala2Traits, // Expand traits defined in Scala 2.11 to simulate old-style rewritings
   new ResolveSuper,        // Implement super accessors and add forwarders to trait methods
   new ArrayConstructors),  // Intercept creation of (non-generic) arrays and intrinsify.

[liufengyun]

Thanks @odersky for the tip, I self-assigned the issue and will look into it once I'm done with the patmat check.

[DarkDimius]

The patmat may be generating a very deep tree in this case.
Simple way to solve this would be to pass -Xss 5M to JVM. I'd go with this way.
The patmat can also be changed not to nest DefDefs that define different cases into each other and instead have all them in the same level. I'd prefer not to do this, as it would may make common subexpression elimination harder to perform. Currently, even very simple CSE that has simple scoping rules can optimize our matches.

[liufengyun]

As @DarkDimius commented, ./bin/dotc -J-Xss5M tests/patmat/patmatexhaust-huge.scala compiles without exception. Then I close this issue.