Rollup merge of rust-lang#107116 - ozkanonur:consolidate-bootstrap-docs, r=jyn514

consolidate bootstrap docs

With this diff, I tried to consolidate bootstrap documentations and remove the duplicated informations.

Coupled with rust-lang/rustc-dev-guide#1563

Resolves rust-lang#90686

Signed-off-by: ozkanonur <[email protected]>

Author: matthiaskrgr

src/bootstrap/README.md

@@ -4,105 +4,31 @@ This is an in-progress README which is targeted at helping to explain how Rust
 is bootstrapped and in general, some of the technical details of the build
 system.
 
-## Using rustbuild
+Note that this README only covers internal information, not how to use the tool.
+Please check [bootstrapping dev guide][bootstrapping-dev-guide] for further information.
 
-The rustbuild build system has a primary entry point, a top level `x.py` script:
+[bootstrapping-dev-guide]: https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html
 
-```sh
-$ python ./x.py build
-```
-
-Note that if you're on Unix, you should be able to execute the script directly:
-
-```sh
-$ ./x.py build
-```
-
-The script accepts commands, flags, and arguments to determine what to do:
-
-* `build` - a general purpose command for compiling code. Alone, `build` will
-  bootstrap the entire compiler, and otherwise, arguments passed indicate what to
-  build. For example:
-
-  ```
-  # build the whole compiler
-  ./x.py build --stage 2
-
-  # build the stage1 compiler
-  ./x.py build
-
-  # build stage0 libstd
-  ./x.py build --stage 0 library/std
-
-  # build a particular crate in stage0
-  ./x.py build --stage 0 library/test
-  ```
-
-  If files that would normally be rebuilt from stage 0 are dirty, the rebuild can be
-  overridden using `--keep-stage 0`. Using `--keep-stage n` will skip all steps
-  that belong to stage n or earlier:
-
-  ```
-  # build stage 1, keeping old build products for stage 0
-  ./x.py build --keep-stage 0
-  ```
-
-* `test` - a command for executing unit tests. Like the `build` command, this
-  will execute the entire test suite by default, and otherwise, it can be used to
-  select which test suite is run:
-
-  ```
-  # run all unit tests
-  ./x.py test
-
-  # execute tool tests
-  ./x.py test tidy
-
-  # execute the UI test suite
-  ./x.py test tests/ui
-
-  # execute only some tests in the UI test suite
-  ./x.py test tests/ui --test-args substring-of-test-name
-
-  # execute tests in the standard library in stage0
-  ./x.py test --stage 0 library/std
-
-  # execute tests in the core and standard library in stage0,
-  # without running doc tests (thus avoid depending on building the compiler)
-  ./x.py test --stage 0 --no-doc library/core library/std
+## Introduction
 
-  # execute all doc tests
-  ./x.py test src/doc
-  ```
+The build system defers most of the complicated logic managing invocations
+of rustc and rustdoc to Cargo itself. However, moving through various stages
+and copying artifacts is still necessary for it to do. Each time rustbuild
+is invoked, it will iterate through the list of predefined steps and execute
+each serially in turn if it matches the paths passed or is a default rule.
+For each step rustbuild relies on the step internally being incremental and
+parallel. Note, though, that the `-j` parameter to rustbuild gets forwarded
+to appropriate test harnesses and such.
 
-* `doc` - a command for building documentation. Like above, can take arguments
-  for what to document.
-
-## Configuring rustbuild
-
-rustbuild offers a TOML-based configuration system with a `config.toml`
-file. An example of this configuration can be found at `config.toml.example`,
-and the configuration file can also be passed as `--config path/to/config.toml`
-if the build system is being invoked manually (via the python script).
-
-You can generate a config.toml using `./configure` options if you want to automate creating the file without having to edit it.
-
-Finally, rustbuild makes use of the [cc-rs crate] which has [its own
-method][env-vars] of configuring C compilers and C flags via environment
-variables.
-
-[cc-rs crate]: https://github.com/alexcrichton/cc-rs
-[env-vars]: https://github.com/alexcrichton/cc-rs#external-configuration-via-environment-variables
-
-## Build stages
+## Build phases
 
 The rustbuild build system goes through a few phases to actually build the
 compiler. What actually happens when you invoke rustbuild is:
 
-1. The entry point script, `x.py` is run. This script is
-   responsible for downloading the stage0 compiler/Cargo binaries, and it then
-   compiles the build system itself (this folder). Finally, it then invokes the
-   actual `bootstrap` binary build system.
+1. The entry point script(`x` for unix like systems, `x.ps1` for windows systems,
+   `x.py` cross-platform) is run. This script is responsible for downloading the stage0
+   compiler/Cargo binaries, and it then compiles the build system itself (this folder).
+   Finally, it then invokes the actual `bootstrap` binary build system.
 2. In Rust, `bootstrap` will slurp up all configuration, perform a number of
    sanity checks (whether compilers exist, for example), and then start building the
    stage0 artifacts.
@@ -115,24 +41,6 @@ compiler. What actually happens when you invoke rustbuild is:
 The goal of each stage is to (a) leverage Cargo as much as possible and failing
 that (b) leverage Rust as much as possible!
 
-## Incremental builds
-
-You can configure rustbuild to use incremental compilation with the
-`--incremental` flag:
-
-```sh
-$ ./x.py build --incremental
-```
-
-The `--incremental` flag will store incremental compilation artifacts
-in `build/<host>/stage0-incremental`. Note that we only use incremental
-compilation for the stage0 -> stage1 compilation -- this is because
-the stage1 compiler is changing, and we don't try to cache and reuse
-incremental artifacts across different versions of the compiler.
-
-You can always drop the `--incremental` to build as normal (but you
-will still be using the local nightly as your bootstrap).
-
 ## Directory Layout
 
 This build system houses all output under the `build` directory, which looks
@@ -236,63 +144,31 @@ build/
     # system will link (using hard links) output from stageN-{std,rustc} into
     # each of these directories.
     #
-    # In theory, there is no extra build output in these directories.
+    # In theory these are working rustc sysroot directories, meaning there is
+    # no extra build output in these directories.
     stage1/
     stage2/
     stage3/
 ```
 
-## Cargo projects
-
-The current build is unfortunately not quite as simple as `cargo build` in a
-directory, but rather the compiler is split into three different Cargo projects:
-
-* `library/std` - the standard library
-* `library/test` - testing support, depends on libstd
-* `compiler/rustc` - the actual compiler itself
-
-Each "project" has a corresponding Cargo.lock file with all dependencies, and
-this means that building the compiler involves running Cargo three times. The
-structure here serves two goals:
-
-1. Facilitating dependencies coming from crates.io. These dependencies don't
-   depend on `std`, so libstd is a separate project compiled ahead of time
-   before the actual compiler builds.
-2. Splitting "host artifacts" from "target artifacts". That is, when building
-   code for an arbitrary target, you don't need the entire compiler, but you'll
-   end up needing libraries like libtest that depend on std but also want to use
-   crates.io dependencies. Hence, libtest is split out as its own project that
-   is sequenced after `std` but before `rustc`. This project is built for all
-   targets.
-
-There is some loss in build parallelism here because libtest can be compiled in
-parallel with a number of rustc artifacts, but in theory, the loss isn't too bad!
-
-## Build tools
-
-We've actually got quite a few tools that we use in the compiler's build system
-and for testing. To organize these, each tool is a project in `src/tools` with a
-corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having
-independent `Cargo.lock` files) and do not currently explicitly depend on the
-compiler or standard library. Compiling each tool is sequenced after the
-appropriate libstd/libtest/librustc compile above.
-
 ## Extending rustbuild
 
-So, you'd like to add a feature to the rustbuild build system or just fix a bug.
-Great! One of the major motivational factors for moving away from `make` is that
-Rust is in theory much easier to read, modify, and write. If you find anything
-excessively confusing, please open an issue on this, and we'll try to get it
-documented or simplified, pronto.
+When you use the bootstrap system, you'll call it through the entry point script
+(`x`, `x.ps1`, or `x.py`). However, most of the code lives in `src/bootstrap`.
+`bootstrap` has a difficult problem: it is written in Rust, but yet it is run
+before the Rust compiler is built! To work around this, there are two components
+of bootstrap: the main one written in rust, and `bootstrap.py`. `bootstrap.py`
+is what gets run by entry point script. It takes care of downloading the `stage0`
+compiler, which will then build the bootstrap binary written in Rust.
 
-First up, you'll probably want to read over the documentation above, as that'll
-give you a high level overview of what rustbuild is doing. You also probably
-want to play around a bit yourself by just getting it up and running before you
-dive too much into the actual build system itself.
+Because there are two separate codebases behind `x.py`, they need to
+be kept in sync. In particular, both `bootstrap.py` and the bootstrap binary
+parse `config.toml` and read the same command line arguments. `bootstrap.py`
+keeps these in sync by setting various environment variables, and the
+programs sometimes have to add arguments that are explicitly ignored, to be
+read by the other.
 
-After that, each module in rustbuild should have enough documentation to keep
-you up and running. Some general areas that you may be interested in modifying
-are:
+Some general areas that you may be interested in modifying are:
 
 * Adding a new build tool? Take a look at `bootstrap/tool.rs` for examples of
   other tools.
@@ -320,8 +196,9 @@ A 'major change' includes
 Changes that do not affect contributors to the compiler or users
 building rustc from source don't need an update to `VERSION`.
 
-If you have any questions, feel free to reach out on the `#t-infra` channel in
-the [Rust Zulip server][rust-zulip] or ask on internals.rust-lang.org. When
-you encounter bugs, please file issues on the rust-lang/rust issue tracker.
+If you have any questions, feel free to reach out on the `#t-infra/bootstrap` channel
+at [Rust Bootstrap Zulip server][rust-bootstrap-zulip]. When you encounter bugs,
+please file issues on the [Rust issue tracker][rust-issue-tracker].
 
-[rust-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra
+[rust-bootstrap-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/t-infra.2Fbootstrap
+[rust-issue-tracker]: https://github.com/rust-lang/rust/issues

src/bootstrap/lib.rs

@@ -11,93 +11,6 @@
 //!   crates.io and Cargo.
 //! * A standard interface to build across all platforms, including MSVC
 //!
-//! ## Architecture
-//!
-//! The build system defers most of the complicated logic managing invocations
-//! of rustc and rustdoc to Cargo itself. However, moving through various stages
-//! and copying artifacts is still necessary for it to do. Each time rustbuild
-//! is invoked, it will iterate through the list of predefined steps and execute
-//! each serially in turn if it matches the paths passed or is a default rule.
-//! For each step rustbuild relies on the step internally being incremental and
-//! parallel. Note, though, that the `-j` parameter to rustbuild gets forwarded
-//! to appropriate test harnesses and such.
-//!
-//! Most of the "meaty" steps that matter are backed by Cargo, which does indeed
-//! have its own parallelism and incremental management. Later steps, like
-//! tests, aren't incremental and simply run the entire suite currently.
-//! However, compiletest itself tries to avoid running tests when the artifacts
-//! that are involved (mainly the compiler) haven't changed.
-//!
-//! When you execute `x.py build`, the steps executed are:
-//!
-//! * First, the python script is run. This will automatically download the
-//!   stage0 rustc and cargo according to `src/stage0.json`, or use the cached
-//!   versions if they're available. These are then used to compile rustbuild
-//!   itself (using Cargo). Finally, control is then transferred to rustbuild.
-//!
-//! * Rustbuild takes over, performs sanity checks, probes the environment,
-//!   reads configuration, and starts executing steps as it reads the command
-//!   line arguments (paths) or going through the default rules.
-//!
-//!   The build output will be something like the following:
-//!
-//!   Building stage0 std artifacts
-//!   Copying stage0 std
-//!   Building stage0 test artifacts
-//!   Copying stage0 test
-//!   Building stage0 compiler artifacts
-//!   Copying stage0 rustc
-//!   Assembling stage1 compiler
-//!   Building stage1 std artifacts
-//!   Copying stage1 std
-//!   Building stage1 test artifacts
-//!   Copying stage1 test
-//!   Building stage1 compiler artifacts
-//!   Copying stage1 rustc
-//!   Assembling stage2 compiler
-//!   Uplifting stage1 std
-//!   Uplifting stage1 test
-//!   Uplifting stage1 rustc
-//!
-//! Let's disect that a little:
-//!
-//! ## Building stage0 {std,test,compiler} artifacts
-//!
-//! These steps use the provided (downloaded, usually) compiler to compile the
-//! local Rust source into libraries we can use.
-//!
-//! ## Copying stage0 {std,test,rustc}
-//!
-//! This copies the build output from Cargo into
-//! `build/$HOST/stage0-sysroot/lib/rustlib/$ARCH/lib`. FIXME: this step's
-//! documentation should be expanded -- the information already here may be
-//! incorrect.
-//!
-//! ## Assembling stage1 compiler
-//!
-//! This copies the libraries we built in "building stage0 ... artifacts" into
-//! the stage1 compiler's lib directory. These are the host libraries that the
-//! compiler itself uses to run. These aren't actually used by artifacts the new
-//! compiler generates. This step also copies the rustc and rustdoc binaries we
-//! generated into build/$HOST/stage/bin.
-//!
-//! The stage1/bin/rustc is a fully functional compiler, but it doesn't yet have
-//! any libraries to link built binaries or libraries to. The next 3 steps will
-//! provide those libraries for it; they are mostly equivalent to constructing
-//! the stage1/bin compiler so we don't go through them individually.
-//!
-//! ## Uplifting stage1 {std,test,rustc}
-//!
-//! This step copies the libraries from the stage1 compiler sysroot into the
-//! stage2 compiler. This is done to avoid rebuilding the compiler; libraries
-//! we'd build in this step should be identical (in function, if not necessarily
-//! identical on disk) so there's no need to recompile the compiler again. Note
-//! that if you want to, you can enable the full-bootstrap option to change this
-//! behavior.
-//!
-//! Each step is driven by a separate Cargo project and rustbuild orchestrates
-//! copying files between steps and otherwise preparing for Cargo to run.
-//!
 //! ## Further information
 //!
 //! More documentation can be found in each respective module below, and you can