add overview

Author: mark-i-m

src/SUMMARY.md

@@ -34,6 +34,7 @@
         - [LLVM ICE-breakers](ice-breaker/llvm.md)
     - [Licenses](./licenses.md)
 - [Part 2: How rustc works](./part-2-intro.md)
+    - [Overview of `rustc`](./overview.md)
     - [High-level overview of the compiler source](./high-level-overview.md)
     - [The Rustc Driver and Interface](./rustc-driver.md)
         - [Rustdoc](./rustdoc.md)

src/overview.md

@@ -0,0 +1,327 @@
+# Rust compiler Overview
+
+This chapter is about the overall process of compiling a program -- how
+everything fits together.
+
+The rust compiler is special in two ways: it does things to your code that
+other compilers don't do (e.g. borrow checking) and it has a lot of
+unconventional implementation choices (e.g. queries). We will talk about these
+in turn in this chapter, and in the rest of the guide, we will look at all the
+individual pieces in more detail.
+
+## What the compiler does to your code
+
+So first, let's look at what the compiler does to your code. For now, we will
+avoid mentioning how the compiler implements these steps except as needed;
+we'll talk about that later.
+
+**TODO: Would be great to have a diagram of this once we nail down the details...**
+
+**TODO: someone else should confirm this vvv**
+
+- User writes a program and invokes `rustc` on it (possibly through `cargo`).
+- First, we parse command line flags, etc. This is done in [`librustc_driver`].
+  We now know what the exact work is we need to do (e.g. which nightly features
+  are enabled, whether we are doing a `check`-only build or emiting LLVM-IR or
+  a full compilation).
+- Then, we start to do compilation...
+- We first [_lex_ the user program][lex]. This turns the program into a stream
+  of _tokens_ (yes, the same sort of tokens as `proc_macros` (sort of)).
+  [`StringReader`] from [`librustc_parse`] integrates [`librustc_lexer`] with
+  `rustc` data structures.
+- We then [_parse_ the stream of tokens][parser] to build an Abstract Syntax
+  Tree (AST).
+- We then take the AST and [convert it to High-Level Intermediate
+  Representation (HIR)][hir]. This is a compiler-friendly representation of the
+  AST.  This involves a lot of desugaring of things like loops and `async fn`.
+- We use the HIR to do [type inference]. This is the process of automatic
+  detection of the type of an expression. **TODO: how `ty` module fits in
+  here**
+- **TODO: Maybe some other things are done here? I think initial type checking
+  happens here? And trait solving?**
+- The HIR is then [lowered to Mid-Level Intermediate Representation (MIR)][mir].
+- The MIR is used for [borrow checking].
+- **TODO: const eval fits in somewhere here I think**
+- We (want to) do [many optimizations on the MIR][mir-opt] because it is still
+  generic and that improves the code we generate later, improving compilation
+  speed too. (**TODO: size optimizations too?**)
+    - MIR is a higher level (and generic) representation, so it is easier to do
+      some optimizations at MIR level than at LLVM-IR level. For example LLVM
+      doesn't seem to be able to optimize the pattern the [`simplify_try`] mir
+      opt looks for.
+- Rust code is _monomorphized_, which means making copies of all the generic
+  code with the type parameters replaced by concrete types. In order to do
+  this, we need to collect a list of what concrete types to generate code for.
+  This is called _monomorphization collection_.
+- We then begin what is vaguely called _code generation_ or _codegen_.
+    - The [code generation stage (codegen)][codegen] is when higher level
+      representations of source are turned into an executable binary. `rustc`
+      uses LLVM for code generation.  The first step is the MIR is then
+      converted to LLVM Intermediate Representation (LLVM IR). This is where
+      the MIR is actually monomorphized, according to the list we created in
+      the previous step.
+    - The LLVM IR is passed to LLVM, which does a lot more optimizations on it.
+      It then emits machine code. It is basically assembly code with additional
+      low-level types and annotations added. (e.g. an ELF object or wasm).
+      **TODO: reference for this section?**
+    - The different libraries/binaries are linked together to produce the final
+      binary. **TODO: reference for this section?**
+
+[`librustc_lexer`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lexer/index.html
+[`librustc_driver`]: https://rust-lang.github.io/rustc-guide/rustc-driver.html
+[lex]: https://rust-lang.github.io/rustc-guide/the-parser.html
+[`StringReader`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/lexer/struct.StringReader.html
+[`librustc_parse`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/index.html
+[parser]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lexer/index.html
+[hir]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lexer/index.html
+[type inference]: https://rust-lang.github.io/rustc-guide/type-inference.html
+[mir]: https://rust-lang.github.io/rustc-guide/mir/index.html
+[borrow checker]: https://rust-lang.github.io/rustc-guide/borrow_check.html
+[mir-opt]: https://rust-lang.github.io/rustc-guide/mir/optimizations.html
+[`simplify_try`]: https://github.com/rust-lang/rust/pull/66282
+[codegen]: https://rust-lang.github.io/rustc-guide/codegen.html
+
+## How it does it
+
+Ok, so now that we have a high-level view of what the compiler does to your
+code, let's take a high-level view of _how_ it does all that stuff. There are a
+lot of constraints and conflicting goals that the compiler needs to
+satisfy/optimize for. For example,
+
+- Compilation speed: how fast is it to compile a program. More/better
+  compile-time analyses often means compilation is slower.
+    - Also, we want to support incremental compilation, so we need to take that
+      into account. How can we keep track of what work needs to be redone and
+      what can be reused if the user modifies their program?
+        - Also we can't store too much stuff in the incremental cache because
+          it would take a long time to load from disk and it could take a lot
+          of space on the user's system...
+- Compiler memory usage: while compiling a program, we don't want to use more
+  memory than we need.
+- Program speed: how fast is your compiled program. More/better compile-time
+  analyses often means the compiler can do better optimizations.
+- Program size: how large is the compiled binary? Similar to the previous
+  point.
+- Compiler compilation speed: how long does it take to compile the compiler?
+  This impacts contributors and compiler maintenance.
+- Compiler implementation complexity: building a compiler is one of the hardest
+  things a person/group can do, and rust is not a very simple language, so how
+  do we make the compiler's code base manageable?
+- Compiler correctness: the binaries produced by the compiler should do what
+  the input programs says they do, and should continue to do so despite the
+  tremendous amount of change constantly going on.
+- Compiler integration: a number of other tools need to use the compiler in
+  various ways (e.g. cargo, clippy, miri, RLS) that must be supported.
+- Compiler stability: the compiler should not crash or fail ungracefully on the
+  stable channel.
+- Rust stability: the compiler must respect rust's stability guarantees by not
+  breaking programs that previously compiled despite the many changes that are
+  always going on to its implementation.
+- Limitations of other tools: rustc uses LLVM in its backend, and LLVM has some
+  strengths we leverage and some limitations/weaknesses we need to work around.
+- And others that I'm probably forgetting.
+
+So, as you read through the rest of the guide, keep these things in mind. They
+will often inform decisions that we make.
+
+### Constant change
+
+One thing to keep in mind is that `rustc` is a real production-quality product.
+As such, it has its fair share of codebase churn and technical debt. A lot of
+the designs discussed throughout this guide are idealized designs that are not
+fully realized yet. And things keep changing so that it is hard to keep this
+guide completely up to date on everything!
+
+The compiler definitely has rough edges, but because of its design it is able
+to keep up with the requirements above.
+
+### Intermediate representations
+
+As with most compilers, `rustc` uses some intermediate representations (IRs) to
+facilitate computations. In general, working directly with the source code is
+extremely inconvenient. Source code is designed to be human-friendly while at
+the same time being unambiguous, but it's less convenient for doing something
+like, say, type checking.
+
+Instead most compilers, including `rustc`, build some sort of IR out of the
+source code which is easier to analyze. `rustc` has a few IRs, each optimized
+for different things:
+
+- Abstract Syntax Tree (AST): the abstract syntax tree is built from the stream
+  of tokens produced by the lexer directly from the source code. It represents
+  pretty much exactly what the user wrote. It helps to do some syntactic sanity
+  checking (e.g. checking that a type is expected where the user wrote one).
+- High-level IR (HIR): This is a sort of very desugared AST. It's still close
+  to what the user wrote syntactically, but it includes some implicit things
+  such as some elided lifetimes, etc. This IR is amenable to type checking.
+- HAIR: This is an intermediate between HIR and MIR. This only exists to make
+  it easier to lower HIR to MIR.
+- Middle-level IR (MIR): This IR is basically a Control-Flow Graph (CFG). A CFG
+  is a type of diagram that shows the basic blocks of a program and how control
+  flow can go between them. Likewise, MIR also has a bunch of basic blocks with
+  simple typed statements inside them (e.g. assignment, simple computations,
+  dropping values, etc). MIR is used for borrow checking and a bunch of other
+  important dataflow based checks, such as checking for uninitialized values.
+  It is also used for a bunch of optimizations and for constant evaluation (via
+  MIRI). Because MIR is still generic, we can do a lot of analyses here more
+  efficiently than after monomorphization.
+- LLVM IR: This is the standard form of all input to the LLVM compiler. LLVM IR
+  is basically a sort of typed assembly language with lots of annotations. It's
+  a standard format that is used by all compilers that use LLVM (e.g. the clang
+  C compiler also outputs LLVM IR). LLVM IR is designed to be easy for other
+  compilers to emit and also rich enough for LLVM to run a bunch of
+  optimizations on it.
+
+### Queries
+
+The first big implementation choice is the _query_ system. The rust compiler
+uses a query system which is unlike most textbook compilers, which are
+organized as a series of passes over the code that execute sequentially. The
+compiler does this to make incremental compilation possible -- that is, if the
+user makes a change to their program and recompiles, we want to do as little
+redundant work as possible to produce the new binary.
+
+In rustc, all the major steps above are organized as a bunch of queries that
+call each other. For example, there is a query to ask for the type of something
+and another to ask for the optimized MIR of a function, and so on. These
+queries can call each other and are all tracked through the query system, and
+the results of the queries are cached on disk so that we can tell which
+queries' results changed from the last compilation and only redo those. This is
+how incremental compilation works.
+
+In principle, for the query-fied steps, we do each of the above for each item
+individually. For example, we will take the HIR for a function and use queries
+to ask for the LLVM IR for that HIR. This drives the generation of optimized
+MIR, which drives the borrow checker, which drives the generation of MIR, and
+so on.
+
+... except that this is very over-simplified. In fact, some queries are not
+cached on disk, and some parts of the compiler have to run for all code anyway
+for correctness even if the code is dead code (e.g. the borrow checker). For
+example, [currently the `mir_borrowck` query is first executed on all functions
+of a crate.][passes] Then the codegen backend invokes the
+`collect_and_partition_mono_items` query, which first recursively requests the
+`optimized_mir` for all reachable functions, which in turn runs `mir_borrowck`
+for that function and then creates codegen units. This kind of split will need
+to remain to ensure that unreachable functions still have their errors emitted.
+
+[passes]: https://github.com/rust-lang/rust/blob/45ebd5808afd3df7ba842797c0fcd4447ddf30fb/src/librustc_interface/passes.rs#L824
+
+Moreover, the compiler wasn't originally built to use a query system; the query
+system has been retrofitted into the compiler, so parts of it are not
+query-fied yet. Also, LLVM isn't our code, so obviously that isn't querified
+either. The plan is to eventually query-fy all of the steps listed in the
+previous section, but as of this writing, only the steps between HIR and
+LLVM-IR are query-fied. That is, lexing and parsing are done all at once for
+the whole program.
+
+One other thing to mention here is the all-important "typing context",
+[`TyCtxt`], which is a giant struct that is at the center of all things. All
+queries are defined as methods on the [`TyCtxt`] type, and the in-memory query
+cache is stored there too. In the code, there is usually a variable called
+`tcx` which is a handle on the typing context. You will also see lifetimes with
+the name `'tcx`, which means that something is tied to the lifetime of the
+`TyCtxt` (usually it is stored or _interned_ there).
+
+[`TyCtxt`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/struct.TyCtxt.html
+
+### `ty::Ty`
+
+Types are really important in Rust, and they form the core of a lot of compiler
+analyses. The main type (in the compiler) that represents types (in the user's
+program) is [`rustc::ty::Ty`][ty]. This is so important that we have a whole chapter
+on [`ty::Ty`][ty], but for now, we just want to mention that it exists and is the way
+`rustc` represents types!
+
+Oh, and also the `rustc::ty` module defines the `TyCtxt` struct we mentioned before.
+
+[ty]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ty/type.Ty.html
+
+### Parallelism
+
+Compiler performance is a problem that we would very much like to improve on
+(and are always working on). One aspect of that is attempting to parallelize
+`rustc` itself.
+
+Currently, there is only one part of rustc that is already parallel: codegen.
+During monomorphization, the compiler will split up all the code to be
+generated into smaller chunks called _codegen units_. These are then generated
+by independent instances of LLVM. Since they are independent, we can run them
+in parallel. At the end, the linker is run to combine all the codegen units
+together into one binary.
+
+However, the rest of the compiler is still not yet parallel. There have been
+lots of efforts spent on this, but it is generally a hard problem. The current
+approach is (**TODO: verify**) to turn `RefCell`s into `Mutex`s -- that is, we
+switch to thread-safe internal mutability. However, there are ongoing
+challenges with lock contention, maintaining query-system invariants under
+concurrency, and the complexity of the code base. One can try out the current
+work by enabling parallel compilation in `config.toml`. It's still early days,
+but there are already some promising performance improvements.
+
+### Bootstrapping
+
+**TODO (or do we want such a section)?**
+
+## A flow chart or walkthrough diagram
+
+**TODO**
+
+# Unresolved Questions
+
+**TODO: find answers to these**
+
+- Does LLVM ever do optimizations in debug builds?
+- How do I explore phases of the compile process in my own sources (lexer,
+  parser, HIR, etc)? - e.g., `cargo rustc -- -Zunpretty=hir-tree` allows you to
+  view HIR representation
+- What is the main source entry point for `X`?
+- Where do phases diverge for cross-compilation to machine  code across
+  different platforms?
+
+# References
+
+- Command line parsing
+    - Guide: [The Rustc Driver and Interface](https://rust-lang.github.io/rustc-guide/rustc-driver.html)
+    - Driver definition: [`rustc_driver`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/)
+    - Main entry point: **TODO**
+- Lexical Analysis: Lex the user program to a stream of tokens
+    - Guide: [Lexing and Parsing](https://rust-lang.github.io/rustc-guide/the-parser.html)
+    - Lexer definition: [`librustc_lexer`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lexer/index.html)
+    - Main entry point: **TODO**
+- Parsing: Parse the stream of tokens to an Abstract Syntax Tree (AST)
+    - Guide: [Lexing and Parsing](https://rust-lang.github.io/rustc-guide/the-parser.html)
+    - Parser definition: [`librustc_parse`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/index.html)
+    - Main entry point: **TODO**
+    - AST definition: [`syntax`](https://doc.rust-lang.org/nightly/nightly-rustc/syntax/index.html)
+- The High Level Intermediate Representation (HIR)
+    - Guide: [The HIR](https://rust-lang.github.io/rustc-guide/hir.html)
+    - Guide: [Identifiers in the HIR](https://rust-lang.github.io/rustc-guide/hir.html#identifiers-in-the-hir)
+    - Guide: [The HIR Map](https://rust-lang.github.io/rustc-guide/hir.html#the-hir-map)
+    - Guide: [Lowering AST to HIR](https://rust-lang.github.io/rustc-guide/lowering.html)
+    - How to view HIR representation for your code `cargo rustc -- -Zunpretty=hir-tree`
+    - Rustc HIR definition: [`rustc_hir`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/index.html)
+    - Main entry point: **TODO**
+- Type Inference
+    - Guide: [Type Inference](https://rust-lang.github.io/rustc-guide/type-inference.html)
+    - Guide: [The ty Module: Representing Types](https://rust-lang.github.io/rustc-guide/ty.html) (semantics)
+    - Main entry point: **TODO**
+- The Mid Level Intermediate Representation (MIR)
+    - Guide: [The MIR (Mid level IR)](https://rust-lang.github.io/rustc-guide/mir/index.html)
+    - Definition: [`librustc/mir`](https://github.com/rust-lang/rust/tree/master/src/librustc/mir)
+    - Definition of source that manipulates the MIR: [`librustc_mir`](https://github.com/rust-lang/rust/tree/master/src/librustc_mir)
+    - Main entry point: **TODO**
+- The Borrow Checker
+    - Guide: [MIR Borrow Check](https://rust-lang.github.io/rustc-guide/borrow_check.html)
+    - Definition: [`rustc_mir/borrow_check`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/borrow_check/index.html)
+    - Main entry point: **TODO**
+- MIR Optimizations
+    - Guide: [MIR Optimizations](https://rust-lang.github.io/rustc-guide/mir/optimizations.html)
+    - Definition: [`rustc_mir/transform`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir/transform/index.html) **TODO: is this correct?**
+    - Main entry point: **TODO**
+- Code Generation
+    - Guide: [Code Generation](https://rust-lang.github.io/rustc-guide/codegen.html)
+    - Guide: [Generating LLVM IR](https://rust-lang.github.io/rustc-guide/codegen.html#generating-llvm-ir) - **TODO: this is not available yet**
+    - Generating Machine Code from LLVM IR with LLVM - **TODO: reference?**
+    - Main entry point MIR -> LLVM IR: **TODO**
+    - Main entry point LLVM IR -> Machine Code **TODO**