extend bootstrap related documentations

Signed-off-by: ozkanonur <[email protected]>

Author: onur-ozkan

src/building/bootstrapping.md

@@ -18,8 +18,23 @@ rustc, then uses it to compile the new compiler.
 
 ## Stages of bootstrapping
 
-Compiling `rustc` is done in stages. Here's a diagram, adapted from Joshua Nelson's
-[talk on bootstrapping][rustconf22-talk] at RustConf 2022, with detailed explanations below.
+The build system defers most of the complicated logic managing invocations
+of rustc and rustdoc to Cargo itself. However, moving through various stages
+and copying artifacts is still necessary for it to do. Each time rustbuild
+is invoked, it will iterate through the list of predefined steps and execute
+each serially in turn if it matches the paths passed or is a default rule.
+For each step rustbuild relies on the step internally being incremental and
+parallel. Note, though, that the `-j` parameter to rustbuild gets forwarded
+to appropriate test harnesses and such.
+
+Most of the "meaty" steps that matter are backed by Cargo, which does indeed
+have its own parallelism and incremental management. Later steps, like
+tests, aren't incremental and simply run the entire suite currently.
+However, compiletest itself tries to avoid running tests when the artifacts
+that are involved (mainly the compiler) haven't changed.
+
+Here's a diagram, adapted from Joshua Nelson's [talk on bootstrapping][rustconf22-talk]
+at RustConf 2022, with detailed explanations below.
 
 The `A`, `B`, `C`, and `D` show the ordering of the stages of bootstrapping.
 <span style="background-color: lightblue; color: black">Blue</span> nodes are downloaded,
@@ -448,3 +463,66 @@ This is an incomplete reference for the outputs generated by bootstrap:
 | copy `rustdoc`                                         | `build/HOST/stage2/bin`                                         |
 
 `--stage=2` stops here.
+
+### Clarification of build command's stdout
+
+In this part, we will investigate the build command's stdout in an action
+(similar, but more detailed and complete documentation compare to topic above).
+When you execute `x.py build` command, the build output will be something
+like the following:
+
+```text
+Building stage0 std artifacts
+Copying stage0 std
+Building stage0 test artifacts
+Copying stage0 test
+Building stage0 compiler artifacts
+Copying stage0 rustc
+Assembling stage1 compiler
+Building stage1 std artifacts
+Copying stage1 std
+Building stage1 test artifacts
+Copying stage1 test
+Building stage1 compiler artifacts
+Copying stage1 rustc
+Assembling stage2 compiler
+Uplifting stage1 std
+Uplifting stage1 test
+Uplifting stage1 rustc
+```
+
+#### Building stage0 {std,test,compiler} artifacts
+
+These steps use the provided (downloaded, usually) compiler to compile the
+local Rust source into libraries we can use.
+
+#### Copying stage0 {std,test,rustc}
+
+This copies the build output from Cargo into
+`build/$HOST/stage0-sysroot/lib/rustlib/$ARCH/lib`.
+[comment]: FIXME: this step's documentation should be expanded -- the information already here may be incorrect.
+
+#### Assembling stage1 compiler
+
+This copies the libraries we built in "building stage0 ... artifacts" into
+the stage1 compiler's lib directory. These are the host libraries that the
+compiler itself uses to run. These aren't actually used by artifacts the new
+compiler generates. This step also copies the rustc and rustdoc binaries we
+generated into `build/$HOST/stage/bin`.
+
+The stage1/bin/rustc is a fully functional compiler, but it doesn't yet have
+any libraries to link built binaries or libraries to. The next 3 steps will
+provide those libraries for it; they are mostly equivalent to constructing
+the stage1/bin compiler so we don't go through them individually.
+
+#### Uplifting stage1 {std,test,rustc}
+
+This step copies the libraries from the stage1 compiler sysroot into the
+stage2 compiler. This is done to avoid rebuilding the compiler; libraries
+we'd build in this step should be identical (in function, if not necessarily
+identical on disk) so there's no need to recompile the compiler again. Note
+that if you want to, you can enable the full-bootstrap option to change this
+behavior.
+
+Each step is driven by a separate Cargo project and rustbuild orchestrates
+copying files between steps and otherwise preparing for Cargo to run.

src/building/how-to-build-and-run.md

@@ -54,12 +54,43 @@ clean` will not cause a rebuild of LLVM.
 
 ## Building the Compiler
 
+Building compiler is unfortunately not quite as simple as building other
+rust projects using`cargo build`. Instead, we use `x.py` script for building
+the compiler. The main reason behind it is the bootstrap stages where you use
+the output of one cargo invocation as the input compiler for another. Which
+is not possible with `cargo` itself.
+
 Note that building will require a relatively large amount of storage space.
 You may want to have upwards of 10 or 15 gigabytes available to build the compiler.
 
-Once you've created a `config.toml`, you are now ready to run
-`x.py`. There are a lot of options here, but let's start with what is
-probably the best "go to" command for building a local compiler:
+The script accepts commands, flags, and arguments to determine what to do.
+To see capabilities of x in an action, check the following examples:
+
+  ```bash
+  # build the whole compiler
+  ./x.py build --stage 2
+
+  # build the stage1 compiler
+  ./x.py build
+
+  # build stage0 libstd
+  ./x.py build --stage 0 library/std
+
+  # build a particular crate in stage0
+  ./x.py build --stage 0 library/test
+  ```
+
+  If files that would normally be rebuilt from stage 0 are dirty, the rebuild can be
+  overridden using `--keep-stage 0`. Using `--keep-stage n` will skip all steps
+  that belong to stage n or earlier:
+
+  ```bash
+  # build stage 1, keeping old build products for stage 0
+  ./x.py build --keep-stage 0
+  ```
+
+Let's dive into details with probably the best "go to"
+command for building a local compiler:
 
 ```bash
 ./x.py build library
@@ -98,6 +129,24 @@ build. The **full** `rustc` build (what you get with `./x.py build
 
 You almost never need to do this.
 
+## Incremental builds
+
+You can configure rustbuild to use incremental compilation with the
+`--incremental` flag:
+
+```sh
+$ ./x.py build --incremental
+```
+
+The `--incremental` flag will store incremental compilation artifacts
+in `build/<host>/stage0-incremental`. Note that we only use incremental
+compilation for the stage0 -> stage1 compilation -- this is because
+the stage1 compiler is changing, and we don't try to cache and reuse
+incremental artifacts across different versions of the compiler.
+
+You can always drop the `--incremental` to build as normal (note that
+you will still be using the downloaded beta as your bootstrap).
+
 ## Build specific components
 
 If you are working on the standard library, you probably don't need to build
@@ -242,3 +291,62 @@ everything up then you only need to run one command!
 
 `rm -rf build` works too, but then you have to rebuild LLVM, which can take
 a long time even on fast computers.
+
+## Build tools
+
+We've actually got quite a few tools that we use in the compiler's build system
+and for testing. To organize these, each tool is a project in `src/tools` with a
+corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having
+independent `Cargo.lock` files) and do not currently explicitly depend on the
+compiler or standard library. Compiling each tool is sequenced after the
+appropriate libstd/libtest/librustc compile above.
+
+## Extending rustbuild
+
+So, you'd like to add a feature to the rustbuild build system or just fix a bug.
+Great! One of the major motivational factors for moving away from `make` is that
+Rust is in theory much easier to read, modify, and write. If you find anything
+excessively confusing, please open an issue on this, and we'll try to get it
+documented or simplified, pronto.
+
+First up, you'll probably want to read over the documentation above, as that'll
+give you a high level overview of what rustbuild is doing. You also probably
+want to play around a bit yourself by just getting it up and running before you
+dive too much into the actual build system itself.
+
+After that, each module in rustbuild should have enough documentation to keep
+you up and running. Some general areas that you may be interested in modifying
+are:
+
+* Adding a new build tool? Take a look at `bootstrap/tool.rs` for examples of
+  other tools.
+* Adding a new compiler crate? Look no further! Adding crates can be done by
+  adding a new directory with `Cargo.toml` followed by configuring all
+  `Cargo.toml` files accordingly.
+* Adding a new dependency from crates.io? This should just work inside the
+  compiler artifacts stage (everything other than libtest and libstd).
+* Adding a new configuration option? You'll want to modify `bootstrap/flags.rs`
+  for command line flags and then `bootstrap/config.rs` to copy the flags to the
+  `Config` struct.
+* Adding a sanity check? Take a look at `bootstrap/sanity.rs`.
+
+If you make a major change, please remember to:
+
++ Update `VERSION` in `src/bootstrap/main.rs`.
+* Update `changelog-seen = N` in `config.toml.example`.
+* Add an entry in `src/bootstrap/CHANGELOG.md`.
+
+A 'major change' includes
+
+* A new option or
+* A change in the default options.
+
+Changes that do not affect contributors to the compiler or users
+building rustc from source don't need an update to `VERSION`.
+
+If you have any questions, feel free to reach out on the `#t-infra` channel in
+the [Rust Zulip server][rust-zulip] or ask on internals.rust-lang.org. When
+you encounter bugs, please file issues on [Rust issue tracker][rust-issue-tracker].
+
+[rust-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra
+[rust-issue-tracker]: https://github.com/rust-lang/rust/issues

src/getting-started.md

@@ -114,6 +114,10 @@ serious development work. In particular, `./x.py build` and `./x.py test`
 provide many ways to compile or test a subset of the code, which can save a lot
 of time.
 
+Also, note that `x.py` supports all kinds of path suffixes for `compiler`, `library`,
+and `src/tools` directories. So, you can simply run `x.py test tidy` instead of
+`x.py test src/tools/tidy`. Or, `x.py build std` instead of `x.py build library/std`.
+
 [rust-analyzer]: ./building/suggested.html#configuring-rust-analyzer-for-rustc
 
 See the chapters on [building](./building/how-to-build-and-run.md),