Full bootstrap: Use Dotty as the reference compiler #5923

smarter · 2019-02-14T16:05:53Z

Before merging this I'd like to:

Wait for 0.13.0-RC1 to be released.
Change the nomenclature used in the build ("non-bootstrapped dotty"
and "bootstrapped dotty" are not descriptive enough when the reference
compiler is also dotty). Will do this later.
Cross-compile the build with 2.12 on the CI so that we don't
accidentally start using Dotty features in case we need to revert this
PR (we'll start usin Dotty features eventually, but let's wait
until we're confident that this setup works well enough).

smarter · 2019-02-14T16:35:09Z

Tasty UUID (001eee9b-f167-8700-005e-a7bc5063cc00) file did not correspond the tasty UUID (00b923fc-a749-ef00-00c0-b993bb014d00) declared in the classfile /tmp/1/library/../out/bootstrap/dotty-library-bootstrapped/scala-0.13/dotty-library_0.13-0.13.0-bin-SNAPSHOT.jar(scala/quoted/package.class).

Hmm, @nicolasstucki do you remember how to debug this ?

nicolasstucki · 2019-02-21T05:44:42Z

I usually printed the tasty files (source file or full tasty) along with the UUID. For this kind of bug, it usually was an older version of the tasty file with a newer class file.

abeln · 2019-03-06T22:52:35Z

project/Build.scala

@@ -30,6 +30,7 @@ import scala.util.Properties.isJavaAtLeast

 object Build {
  val scalacVersion = "2.12.8"
+  val referenceVersion = "0.13.0-bin-20190213-d7090ab-NIGHTLY"


We'll need a more recent nightly, because of quotes in StagedTuple and b32a4a8

abeln · 2019-03-07T18:48:48Z

@nicolasstucki I'm trying to patch this in locally for our explicit nulls work (we want to bootstrap the compiler with explicit nulls), and I'm seeing the same error

Tasty UUID (0087cb62-52b8-0e00-003f-c5611a437a00) file did not correspond the tasty UUID (0074c917-b8c8-e900-00f9-8f1e95ce9c00) declared in the classfile /Users/abeln/src/dotty2/dotty/library/../out/bootstrap/dotty-library-bootstrapped/scala-0.14/dotty-library_0.14-0.14.0-bin-SNAPSHOT.jar(dotty/DottyPredef.class).

When you say

For this kind of bug, it usually was an older version of the tasty file with a newer class file.

does that suggest that the fix is cleaning some sbt cache? Or do I need to do something else? Thanks!

smarter · 2019-03-07T18:56:38Z

There's probably some classpath confusion somewhere, I'll investigate in the near future, but I'm not sure why this is needed for you given that dotty-bootstrapped/compile should already give you the information you want

abeln · 2019-03-07T19:24:41Z

There's probably some classpath confusion somewhere, I'll investigate in the near future, but I'm not sure why this is needed for you given that dotty-bootstrapped/compile should already give you the information you want

Ok, but I see the error when I run dotc, so I'm not able to run the boostrapped compiler on any files. In any case, thanks for looking into this.

smarter · 2019-03-07T19:43:57Z

What error ?

smarter · 2019-03-07T19:45:19Z

To clarify: Iwas saying that you don't need this PR to test the bootstrapped compiler since you can already do one stage of bootstrapping with the current build

abeln · 2019-03-07T19:55:40Z

To clarify: Iwas saying that you don't need this PR to test the bootstrapped compiler since you can already do one stage of bootstrapping with the current build

I'm confused. When we switch to explicit nulls, we'll need to use union types inside dotty. My understanding of the current bootstrapping process is

scalac compiles dotty
dotty compiles dotty

If we use union types inside dotty, then step 1) will fail, because scalac doesn't have union types, right? What am I missing?

smarter · 2019-03-07T20:00:47Z

Ah ok, I see now, but can't you work around that by defining something like:

type Nullable[T] = T | Null

In a file which is only compiled by the bootstrapped compiler, and:

type Nullable[T] = T

in a file that is only compiled by Scala 2. Then everything else can cross-compile.

nicolasstucki · 2019-04-22T06:20:33Z

Both tests/run/i5536/Macro_1.scala and tests/run/i5533b/Macro_1.scala are using implicit val toolbox ... by mistake, if you remove that line you will be able to place them in tests/run-macros.

nicolasstucki

Otherwise LGTM

project/Build.scala

The non-bootstrapped compiler is now compiled by Dotty 0.14.0-RC1 instead of Scala 2.12.8. The bootstrapped compiler is compiled using the non-bootstrapped compiler as before. For now, we still support compilation with Scala 2.12, this is achieved by having separate `src-2.x` and `src-3.x` directory for the library. To use Scala 2.12 as the reference compiler, run: ++2.12.8 in sbt, or change `referenceVersion` to "2.12.8" in Build.scala.

nicolasstucki

Otherwise LGTM

tests/run/i5536/Macro_1.scala

Compiling a macro with a non-bootstrapped compiler is fine, but running that macro in a subsequent compiler run as is usually done in macro tests is problematic since the MacroClassLoader is created using the compiler classpath (which contains a bootstrapped library) with the regular JVM classpath as parent classloader (which contains a non-bootstrapped library), this is inconsistent and could lead to various weird problems depending on the order in which things get loaded. One possible way to run these tests would be to always put the bootstrapped library on the JVM classpath, even if we're using the non-bootstrapped compiler, but this would cause test failures every time we would break binary compatibility. Instead, this commit moves the relevant tests to {pos,neg,run}-macros which will only be run with a bootstrapped compiler. To speed up development, it might be useful to add a flag to `testCompilation` that would put the bootstrapped library on the JVM classpath and still run these tests, but this is left as future work.

Like explained in the previous commit this kind of tests should be bootstrapped-only.

Some check files were not deleted or moved when the corresponding test was deleted or moved.

smarter · 2019-04-22T13:40:30Z

@odersky @anatoliykmetyuk @biboudis @OlivierBlanvillain @liufengyun @AleksanderBG @milessabin @allanrenucci @sjrd This PR is finally in! Here's a few things to watch out for now that we use Dotty instead of Scala 2.12 as our reference compiler:

Incremental compilation issues, if you notice them please open an issue with a branch where the problem can be reproduced:
- Undercompilation: When we make a change, hit compile, and some files that should be recompiled are not. This can manifest itself in various ways such as a MethodNotFoundException at runtime (as an aside, switching to Dotty fixes one long-standing undercompilation issue: if you change the body of a final val, every usage of that final val should now get correctly recompiled).
- Overcompilation: When we make a change, hit compile, and a lot more files than expected end up being recompiled. This is benign but annoying.
Classpath confusion: Especially when running the non-bootstrapped compiler, it's easy to get confused: on the JVM classpath (the one set using java -classpath ... used to load classes in the JVM) we have a non-bootstrapped library and compiler, but on the compiler classpath (the one set using dotc -classpath ... used to create symbols in the compiler) we always have the bootstrapped library (since we may not be able to load symbols from the tasty files generated by the reference compiler). I've tried to make sure we use the correct classpath everywhere but there might still be some problems lurking, they can manifest themselves in various ways such as a "TASTy signature has wrong version." exception.
Things get even more complicated when using macros since running a macro involves loading the compiler classpath into the JVM classpath, therefore macro tests are now limited to the bootstrapped compiler, see aed2582

If at any point you need to go back to using Scala 2.12 as the reference compiler, you can do it very easily by running:

++2.12.8

Inside sbt. Or by changing https://github.com/lampepfl/dotty/blob/b8fddaf23fd1e7101e6fca5f8b45cd9f037179e6/project/Build.scala#L33 to "2.12.8".

Happy hacking!

See scala/scala3#5923

smarter added stat:wip stat:do not merge labels Feb 14, 2019

smarter added this to the 0.14 Tech Preview milestone Feb 14, 2019

abeln reviewed Mar 6, 2019

View reviewed changes

smarter force-pushed the full-bootstrap branch 3 times, most recently from a2dccf6 to 7d1f407 Compare March 26, 2019 15:43

anatoliykmetyuk modified the milestones: 0.14 Tech Preview, 0.15 Tech Preview Apr 11, 2019

smarter changed the title ~~[WIP] Full bootstrap: Use Dotty as the reference compiler~~ Full bootstrap: Use Dotty as the reference compiler Apr 21, 2019

smarter force-pushed the full-bootstrap branch from 7d1f407 to 0a8ad93 Compare April 21, 2019 21:06

smarter requested a review from nicolasstucki April 21, 2019 21:06

smarter assigned nicolasstucki Apr 21, 2019

smarter removed stat:do not merge stat:wip labels Apr 21, 2019

nicolasstucki approved these changes Apr 22, 2019

View reviewed changes

project/Build.scala Show resolved Hide resolved

nicolasstucki assigned smarter and unassigned nicolasstucki Apr 22, 2019

smarter force-pushed the full-bootstrap branch from 0a8ad93 to d75092c Compare April 22, 2019 11:33

nicolasstucki approved these changes Apr 22, 2019

View reviewed changes

tests/run/i5536/Macro_1.scala Show resolved Hide resolved

smarter added 3 commits April 22, 2019 14:01

Move REPL test involving a macro

37fa919

Like explained in the previous commit this kind of tests should be bootstrapped-only.

Delete or move stray .check files

ee7cc53

Some check files were not deleted or moved when the corresponding test was deleted or moved.

smarter force-pushed the full-bootstrap branch from d75092c to ee7cc53 Compare April 22, 2019 12:07

smarter merged commit b8fddaf into scala:master Apr 22, 2019

smarter mentioned this pull request Apr 22, 2019

Bootstrap compiler on Dotty 0.12.0-RC1 #4842

Closed

allanrenucci deleted the full-bootstrap branch April 22, 2019 14:45

smarter mentioned this pull request May 2, 2019

Increase tests timeout #6428

Closed

biboudis mentioned this pull request May 22, 2019

0.15.0-RC1 Blog Post Draft #6544

Merged

romac mentioned this pull request May 24, 2019

Cross-compilation support for Scala 3 (WIP) epfl-lara/inox#71

Closed

18 tasks

smarter added a commit to lampepfl/dotty-ci that referenced this pull request Sep 10, 2019

dotty-docker: add support for test_scala212

25beef3

See scala/scala3#5923

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Full bootstrap: Use Dotty as the reference compiler #5923

Full bootstrap: Use Dotty as the reference compiler #5923

Uh oh!

smarter commented Feb 14, 2019 •

edited

Loading

Uh oh!

smarter commented Feb 14, 2019

Uh oh!

nicolasstucki commented Feb 21, 2019

Uh oh!

abeln Mar 6, 2019

Uh oh!

abeln commented Mar 7, 2019

Uh oh!

smarter commented Mar 7, 2019

Uh oh!

abeln commented Mar 7, 2019

Uh oh!

smarter commented Mar 7, 2019

Uh oh!

smarter commented Mar 7, 2019

Uh oh!

abeln commented Mar 7, 2019 •

edited

Loading

Uh oh!

smarter commented Mar 7, 2019

Uh oh!

nicolasstucki commented Apr 22, 2019

Uh oh!

nicolasstucki left a comment

Uh oh!

Uh oh!

nicolasstucki left a comment

Uh oh!

Uh oh!

smarter commented Apr 22, 2019

Uh oh!

Uh oh!

Full bootstrap: Use Dotty as the reference compiler #5923

Full bootstrap: Use Dotty as the reference compiler #5923

Uh oh!

Conversation

smarter commented Feb 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarter commented Feb 14, 2019

Uh oh!

nicolasstucki commented Feb 21, 2019

Uh oh!

abeln Mar 6, 2019

Choose a reason for hiding this comment

Uh oh!

abeln commented Mar 7, 2019

Uh oh!

smarter commented Mar 7, 2019

Uh oh!

abeln commented Mar 7, 2019

Uh oh!

smarter commented Mar 7, 2019

Uh oh!

smarter commented Mar 7, 2019

Uh oh!

abeln commented Mar 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarter commented Mar 7, 2019

Uh oh!

nicolasstucki commented Apr 22, 2019

Uh oh!

nicolasstucki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nicolasstucki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smarter commented Apr 22, 2019

Uh oh!

Uh oh!

smarter commented Feb 14, 2019 •

edited

Loading

abeln commented Mar 7, 2019 •

edited

Loading