Add tests for trailing comma parsing

[adampauls]

Fixes scala/bug#11900 in dotty. Forward port of scala/scala#8780.

@odersky put up an alternative fix in #14757. This PR just adds tests (some of which are from scala/scala#8780).

[adampauls]

Test pulled over from scala/scala#8780.

[odersky]

I am afraid that confirms my doubt. This pretty much murders the grammar. It seems out of proportion to accept such a drastic complication for such a minor change like this.

In retrospect I think it was a mistake that I even accepted (grudgingly) to add trailing commas in the language. My suspicion was that the cost would be higher than the benefit and that has born out as far as I can see.

[som-snytt]

Trailing commas improves the lives of millions of each day. The lexer compromise may have been most appropriate.

[adampauls]

I'm afraid I don't follow. The current implementation of trailing commas murders the grammar even more. Significant indentation murders the grammar. Yet trailing commas aren't even documented at all in the grammar because they are hidden in the lexer. Significant indentation is documented with illegal EBNF. Unless I'm mistaken, if you held either the original trailing commas SIP or significant indentation to the same standard you are holding this change, they would never have happened.

Would it be alright if I just added a few lines of prose and a special symbol like <<< >>> for comma separated lists?

[adampauls]

I updated the grammar to be more readable and a only sort of abuse notation. I could go even further and define e.g.

<)> ::= [ ',' nl ] ')'

, which I think would be about what you would have to do to document the current behavior anyways.

[som-snytt]

Speaking of a murder of grammars (like crows), I just wondered about trailing commas in optional braces, as I just had a need to write one.

enum X {
  case A,
       B,
       C,
}
enum Y:
  case A,
       B,
       C,
end Y

Have we joked, "comma chameleon" on this thread yet? like the '80s hit song.

[adampauls]

I went ahead and made some specific notation for delimiters in a comma-separated environment.

[odersky]

Whatever is done here is way too complex/expensive for the very small gains it provides. I am convinced this is a dead end.

There IS a way to improve the Scanner to be more discriminating when it accepts commas, since it now actually knows whether commas can be expected. It's a bit fishy since so far this info was only passed for error recover purposes. But i think it's better than trying to change the grammar and the parser. In a sense trailing commas is a hack (yes, that's the right term!) for usability. If we can confine that hack in the Scanner that's better than polluting everything else with it.

[odersky]

I think it would be good to add the tests to #14757 or make a separate PR with them once #14757 is merged.

[adampauls]

Ah, I did not realize you viewed trailing commas in general as a hack -- that explains your reasoning here a bit better. I will just point out the core change is actually 2 fewer lines of code than the current implementation, but I can understand why you might want to hide a hack. I will add the tests once #14757 is merged.

[odersky]

Thanks for the tests. Glad everything works as expected.

[som-snytt]

Thanks, this was educational. I backported the alternative coordination between parser and scanner on trailing comma to Scala 2. The only thing that regressed (edit: so far) was a quasiquote syntax error message that had progressed two years ago.