From 06c5c92c2b5405dadf0360bc2f63ab922eacd68a Mon Sep 17 00:00:00 2001 From: Michael Howell Date: Mon, 13 Mar 2023 13:37:22 -0700 Subject: [PATCH 1/5] Update rustdoc-internals.md --- src/rustdoc-internals.md | 228 ++++++++++++++++++++++++--------------- 1 file changed, 141 insertions(+), 87 deletions(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index a8cc0c376..9c6ffe055 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -7,38 +7,61 @@ see the ["Rustdoc overview" chapter](./rustdoc.md). ## From crate to clean -In `core.rs` are two central items: the `DocContext` struct, and the `run_core` -function. The latter is where rustdoc calls out to rustc to compile a crate to -the point where rustdoc can take over. The former is a state container used -when crawling through a crate to gather its documentation. +In `core.rs` are two central items: the `DocContext` struct, and the +`run_global_ctxt` function. The latter is where rustdoc calls out to rustc to +compile a crate to the point where rustdoc can take over. The former is a state +container used when crawling through a crate to gather its documentation. The main process of crate crawling is done in `clean/mod.rs` through several -implementations of the `Clean` trait defined within. This is a conversion -trait, which defines one method: +functions with names that start with `clean_`. Each function accepts an `hir` +or `ty` data structure, and outputs a `clean` structure used by rustdoc. For +example, this function for converting lifetimes: ```rust,ignore -pub trait Clean { - fn clean(&self, cx: &DocContext) -> T; +fn clean_lifetime<'tcx>(lifetime: &hir::Lifetime, cx: &mut DocContext<'tcx>) -> Lifetime { + let def = cx.tcx.named_bound_var(lifetime.hir_id); + if let Some( + rbv::ResolvedArg::EarlyBound(node_id) + | rbv::ResolvedArg::LateBound(_, _, node_id) + | rbv::ResolvedArg::Free(_, node_id), + ) = def + { + if let Some(lt) = cx.substs.get(&node_id).and_then(|p| p.as_lt()).cloned() { + return lt; + } + } + Lifetime(lifetime.ident.name) } ``` `clean/mod.rs` also defines the types for the "cleaned" AST used later on to -render documentation pages. Each usually accompanies an implementation of -`Clean` that takes some AST or HIR type from rustc and converts it into the +render documentation pages. Each usually accompanies a `clean` function +that takes some AST or HIR type from rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated items may -have some extra processing in its `Clean` implementation, but for the most part +have some extra processing in its `clean` function, but for the most part these impls are straightforward conversions. The "entry point" to this module -is the `impl Clean for visit_ast::RustdocVisitor`, which is called by -`run_core` above. - -You see, I actually lied a little earlier: There's another AST transformation -that happens before the events in `clean/mod.rs`. In `visit_ast.rs` is the -type `RustdocVisitor`, which *actually* crawls a `rustc_hir::Crate` to get the first -intermediate representation, defined in `doctree.rs`. This pass is mainly to -get a few intermediate wrappers around the HIR types and to process visibility -and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and -`#[doc(hidden)]` are processed, as well as the logic for whether a `pub use` -should get the full page or a "Reexport" line in the module page. +is `clean::krate`, which is called by +`run_global_ctxt` above. + +The first step in `clean::krate` is to invoke `visit_ast::RustdocVisitor` to +process the module tree into an intermediate `visit_ast::Module`. This is the +step that actually crawls the `rustc_hir::Crate`, normalizing various aspects +of name resolution, such as: + + * showing `#[macro_export]`-ed macros at the crate root, regardless of where + they're defined + * inlining public `use` exports of private items, or showing a "Reexport" + line in the module page + * inlining items with `#[doc(hidden)]` if the base item is hidden but the + reexport is not + * handling `#[doc(inline)]` and `#[doc(no_inline)]` + * handling import globs and cycles, so there are no duplicates or infinite + directory trees + +After this step, `clean::krate` invokes `clean_doc_module`, which actually +converts the HIR items to the cleaned AST. This is also the step where cross- +crate inlining is performed, which requires converting `rustc_middle` data +structures into the cleaned AST instead. The other major thing that happens in `clean/mod.rs` is the collection of doc comments and `#[doc=""]` attributes into a separate field of the Attributes @@ -48,41 +71,28 @@ easier to collect this documentation later in the process. The primary output of this process is a `clean::Crate` with a tree of Items which describe the publicly-documentable items in the target crate. -### Hot potato +### Passes anything but a gas station + +(alternate title: [hot potato](https://www.youtube.com/watch?v=WNFBIt5HxdY)) Before moving on to the next major step, a few important "passes" occur over -the documentation. These do things like combine the separate "attributes" into -a single string to make the document easier on the markdown parser, -or drop items that are not public or deliberately hidden with `#[doc(hidden)]`. +the cleaned AST. Several of these passes are lints and reports, but some of +them mutate or generate new items. + These are all implemented in the `passes/` directory, one file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. Note that unlike the previous set of AST transformations, the passes are run on the _cleaned_ crate. -(Strictly speaking, you can fine-tune the passes run and even add your own, but -[we're trying to deprecate that][44136]. If you need finer-grain control over -these passes, please let us know!) - -[44136]: https://github.com/rust-lang/rust/issues/44136 - -Here is the list of passes as of November 2022: +Here is the list of passes as of March 2023: - `calculate-doc-coverage` calculates information used for the `--show-coverage` flag. -- `check-bare-urls` detects links that are not linkified, e.g., in Markdown such as - `Go to https://example.com/.` It suggests wrapping the link with angle brackets: - `Go to .` to linkify it. This is the code behind the `rustdoc::bare_urls` lint. - -- `check-code-block-syntax` validates syntax inside Rust code blocks - (```rust) - -- `check-doc-test-visibility` runs doctest visibility–related lints. - -- `check-invalid-html-tags` detects invalid HTML (like an unclosed ``) - in doc comments. +- `check-doc-test-visibility` runs doctest visibility–related lints. This pass + runs before `strip-private`, which is why it needs to be separate from + `run-lints`. - `collect-intra-doc-links` resolves [intra-doc links](https://doc.rust-lang.org/nightly/rustdoc/write-documentation/linking-to-items-by-name.html). @@ -92,44 +102,66 @@ Here is the list of passes as of November 2022: - `propagate-doc-cfg` propagates `#[doc(cfg(...))]` to child items. +- `run-lints` runs some of rustdoc's lints, defind in `passes/lint`. This is + the last pass to run. + + - `bare_urls` detects links that are not linkified, e.g., in Markdown such as + `Go to https://example.com/.` It suggests wrapping the link with angle brackets: + `Go to .` to linkify it. This is the code behind the `rustdoc::bare_urls` lint. + + - `check_code_block_syntax` validates syntax inside Rust code blocks + (```rust) + + - `html_tags` detects invalid HTML (like an unclosed ``) + in doc comments. + +- `strip-hidden` and `strip-private` strip all `doc(hidden)` and private items + from the output. `strip-private` implies `strip-priv-imports`. Basically, the + goal is to remove items that are not relevant for public documentation. This + pass is skipped when `--document-hidden-items` is passed. + - `strip-priv-imports` strips all private import statements (`use`, `extern crate`) from a crate. This is necessary because rustdoc will handle *public* imports by either inlining the item's documentation to the module or creating a "Reexports" section with the import in it. The pass ensures that all of - these imports are actually relevant to documentation. + these imports are actually relevant to documentation. It is technically + only run when `--document-private-items` is passed, but `strip-private` + accomplishes the same thing. -- `strip-hidden` and `strip-private` strip all `doc(hidden)` and private items - from the output. `strip-private` implies `strip-priv-imports`. Basically, the - goal is to remove items that are not relevant for public documentation. +- `strip-private` strips all private items from a crate which cannot be seen + externally. This pass is skipped when `--document-private-items` is passed. There is also a `stripper` module in `passes/`, but it is a collection of utility functions for the `strip-*` passes and is not a pass itself. -## From clean to crate +## From clean to HTML This is where the "second phase" in rustdoc begins. This phase primarily lives -in the `html/` folder, and it all starts with `run()` in `html/render.rs`. This -code is responsible for setting up the `Context`, `SharedContext`, and `Cache` -which are used during rendering, copying out the static files which live in -every rendered set of documentation (things like the fonts, CSS, and JavaScript -that live in `html/static/`), creating the search index, and printing out the -source code rendering, before beginning the process of rendering all the -documentation for the crate. - -Several functions implemented directly on `Context` take the `clean::Crate` and -set up some state between rendering items or recursing on a module's child -items. From here the "page rendering" begins, via an enormous `write!()` call -in `html/layout.rs`. The parts that actually generate HTML from the items and -documentation occurs within a series of `std::fmt::Display` implementations and -functions that pass around a `&mut std::fmt::Formatter`. The top-level -implementation that writes out the page body is the `impl<'a> fmt::Display for -Item<'a>` in `html/render.rs`, which switches out to one of several `item_*` -functions based on the kind of `Item` being rendered. +in the `formats/` and `html/` folders, and it all starts with +`formats::run_format`. This code is responsible for setting up a type that +`impl FormatRenderer`, which for HTML is [`Context`]. + +This structure contains methods that get called by `run_format` to drive the +doc rendering, which includes: + +* `init` generates `static.files`, as well as search index and `src/` +* `item` generates the item HTML files themselves +* `after_krate` generates other global resources like `all.html` + +In `item`, the "page rendering" occurs, via a mixture of [Askama] templates +and manual `write!()` calls, starting in `html/layout.rs`. The parts that have +not been converted to templates occur within a series of `std::fmt::Display` +implementations and functions that pass around a `&mut std::fmt::Formatter`. + +The parts that actually generate HTML from the items and documentation start +with `print_item` defined in `html/render/print_item.rs`, which switches out +to one of several `item_*` functions based on kind of `Item` being rendered. Depending on what kind of rendering code you're looking for, you'll probably -find it either in `html/render.rs` for major items like "what sections should I -print for a struct page" or `html/format.rs` for smaller component pieces like -"how should I print a where clause as part of some other item". +find it either in `html/render/mod.rs` for major items like "what sections +should I print for a struct page" or `html/format/mod.rs` for smaller +component pieces like "how should I print a where clause as part of some other item". Whenever rustdoc comes across an item that should print hand-written documentation alongside, it calls out to `html/markdown.rs` which interfaces @@ -148,23 +180,45 @@ to us"][video]) [video]: https://www.youtube.com/watch?v=hOLAGYmUQV0 -It's important to note that the AST cleaning can ask the compiler for -information (crucially, `DocContext` contains a `TyCtxt`), but page rendering -cannot. The `clean::Crate` created within `run_core` is passed outside the -compiler context before being handed to `html::render::run`. This means that a -lot of the "supplementary data" that isn't immediately available inside an -item's definition, like which trait is the `Deref` trait used by the language, -needs to be collected during cleaning, stored in the `DocContext`, and passed -along to the `SharedContext` during HTML rendering. This manifests as a bunch -of shared state, context variables, and `RefCell`s. - -Also of note is that some items that come from "asking the compiler" don't go -directly into the `DocContext` - for example, when loading items from a foreign -crate, rustdoc will ask about trait implementations and generate new `Item`s -for the impls based on that information. This goes directly into the returned -`Crate` rather than roundabout through the `DocContext`. This way, these -implementations can be collected alongside the others, right before rendering -the HTML. +It's important to note that rustdoc can ask the compiler for type information +directly, even during HTML generation. This [didn't used to be the case], and +a lot of rustdoc's architecture was designed around not doing that, but a +`TyCtxt` is now passed to `formats::renderer::run_format`, which is used to +run generation for both HTML and the (unstable as of March 2023) JSON format. + +[didn't used to be the case]: https://github.com/rust-lang/rust/pull/80090 + +This change has allowed other changes to remove data from the "clean" AST +that can be easily derived from `TyCtxt` queries, and we'll usually accept +PRs that remove fields from "clean" (it's been soft-deprecated), but this +is complicated from two other constraints that rustdoc runs under: + +* Docs can be generated for crates that don't actually pass type checking. + This is used for generating docs that cover mutually-exclusive platform + configurations, such as `libstd` having a single package of docs that + cover all supported operating systems. This means rustdoc has to be able + to generate docs from HIR. +* Docs can inline across crates. Since crate metadata doesn't contain HIR, + it must be possible to generate inlined docs from the `rustc_middle` data. + +The "clean" AST acts as a common output format for both input formats. There +is also some data in clean that doesn't correspond directly to HIR, such as +synthetic `impl`s for auto traits and blanket `impl`s generated by the +`collect-trait-impls` pass. + +Some additional data is stored in +`html::render::context::{Context, SharedContext}`. These two types serve as +ways to segregate rustdoc's data for an eventual future with multithreaded doc +generation, as well as just keeping things organized: + +* [`Context`] stores data used for generating the current page, such as its + path, a list of HTML IDs that have been used (to avoid duplicate `id=""`), + and the pointer to `SharedContext`. +* [`SharedContext`] stores data that does not vary by page, such as the `tcx` + pointer, and a list of all types. + +[`Context`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/html/render/context/struct.Context.html +[`SharedContext`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/html/render/context/struct.SharedContext.html ## Other tricks up its sleeve From c46ce3334caf526a64bc90a612ac730f6026ebe6 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Tue, 14 Mar 2023 05:35:31 +0200 Subject: [PATCH 2/5] typo --- src/rustdoc-internals.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index 9c6ffe055..61fd67b83 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -102,7 +102,7 @@ Here is the list of passes as of March 2023: - `propagate-doc-cfg` propagates `#[doc(cfg(...))]` to child items. -- `run-lints` runs some of rustdoc's lints, defind in `passes/lint`. This is +- `run-lints` runs some of rustdoc's lints, defined in `passes/lint`. This is the last pass to run. - `bare_urls` detects links that are not linkified, e.g., in Markdown such as From a25c13c927ec5bb87d044a9e6de653e8dc5f72e2 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Tue, 14 Mar 2023 05:35:57 +0200 Subject: [PATCH 3/5] fix date-check annotation --- src/rustdoc-internals.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index 61fd67b83..13c337c71 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -108,7 +108,7 @@ Here is the list of passes as of March 2023: - `bare_urls` detects links that are not linkified, e.g., in Markdown such as `Go to https://example.com/.` It suggests wrapping the link with angle brackets: `Go to .` to linkify it. This is the code behind the `rustdoc::bare_urls` lint. + date-check: 2022-05 --> `rustdoc::bare_urls` lint. - `check_code_block_syntax` validates syntax inside Rust code blocks (```rust) From b7d039d88aa72b6463bb051c12c7df59caf36c77 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Tue, 14 Mar 2023 05:37:07 +0200 Subject: [PATCH 4/5] fix date-check annotation some more --- src/rustdoc-internals.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index 13c337c71..8f5d485fe 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -108,7 +108,7 @@ Here is the list of passes as of March 2023: - `bare_urls` detects links that are not linkified, e.g., in Markdown such as `Go to https://example.com/.` It suggests wrapping the link with angle brackets: `Go to .` to linkify it. This is the code behind the `rustdoc::bare_urls` lint. + date-check: may 2022 --> `rustdoc::bare_urls` lint. - `check_code_block_syntax` validates syntax inside Rust code blocks (```rust) From d3f1dd826e63bf7be22badbcd03159dbb5ee1a07 Mon Sep 17 00:00:00 2001 From: Tshepang Mbambo Date: Tue, 14 Mar 2023 05:47:35 +0200 Subject: [PATCH 5/5] alert when date gets stale (by using date-check annotation) --- src/rustdoc-internals.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/rustdoc-internals.md b/src/rustdoc-internals.md index 8f5d485fe..d58c2d280 100644 --- a/src/rustdoc-internals.md +++ b/src/rustdoc-internals.md @@ -184,7 +184,8 @@ It's important to note that rustdoc can ask the compiler for type information directly, even during HTML generation. This [didn't used to be the case], and a lot of rustdoc's architecture was designed around not doing that, but a `TyCtxt` is now passed to `formats::renderer::run_format`, which is used to -run generation for both HTML and the (unstable as of March 2023) JSON format. +run generation for both HTML and the +(unstable as of March 2023) JSON format. [didn't used to be the case]: https://github.com/rust-lang/rust/pull/80090