rustdoc: Further improve chapters and sections on testing

Author: fmease

src/SUMMARY.md

@@ -101,6 +101,8 @@
 - [Rustdoc internals](./rustdoc-internals.md)
     - [Search](./rustdoc-internals/search.md)
 	- [The `rustdoc` test suite](./rustdoc-internals/rustdoc-test-suite.md)
+	- [The `rustdoc-gui` test suite](./rustdoc-internals/rustdoc-gui-test-suite.md)
+	- [The `rustdoc-json` test suite](./rustdoc-internals/rustdoc-json-test-suite.md)
 - [Autodiff internals](./autodiff/internals.md)
     - [Installation](./autodiff/installation.md)
     - [How to debug](./autodiff/debugging.md)

src/rustdoc-internals.md

@@ -270,35 +270,6 @@ in `test.rs` is the function `make_test`, which is where hand-written
 Some extra reading about `make_test` can be found
 [here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/).
 
-## Dotting i's And Crossing t's
-
-So that's `rustdoc`'s code in a nutshell, but there's more things in the
-compiler that deal with it. Since we have the full `compiletest` suite at hand,
-there's a set of tests in `tests/rustdoc` that make sure the final `HTML` is
-what we expect in various situations. These tests also use a supplementary
-script, `src/etc/htmldocck.py`, that allows it to look through the final `HTML`
-using `XPath` notation to get a precise look at the output. The full
-description of all the commands available to `rustdoc` tests (e.g. [`@has`] and
-[`@matches`]) is in [`htmldocck.py`].
-
-To use multiple crates in a `rustdoc` test, add `//@ aux-build:filename.rs`
-to the top of the test file. `filename.rs` should be placed in an `auxiliary`
-directory relative to the test file with the comment. If you need to build
-docs for the auxiliary file, use `//@ build-aux-docs`.
-
-In addition, there are separate tests for the search index and `rustdoc`'s
-ability to query it. The files in `tests/rustdoc-js` each contain a
-different search query and the expected results, broken out by search tab.
-These files are processed by a script in `src/tools/rustdoc-js` and the `Node.js`
-runtime. These tests don't have as thorough of a writeup, but a broad example
-that features results in all tabs can be found in `basic.js`. The basic idea is
-that you match a given `QUERY` with a set of `EXPECTED` results, complete with
-the full item path of each item.
-
-[`@has`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L39
-[`@matches`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L44
-[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py
-
 ## Testing Locally
 
 Some features of the generated `HTML` documentation might require local

src/rustdoc-internals/rustdoc-gui-test-suite.md

@@ -0,0 +1,14 @@
+# The `rustdoc-gui` test suite
+
+> **FIXME**: This section is a stub. Please help us flesh it out!
+
+This page is about the test suite named `rustdoc-gui` used to test the "GUI" of `rustdoc` (i.e., the HTML/JS/CSS as rendered in a browser).
+For other rustdoc-specific test suites, see [Rustdoc test suites].
+
+These use a NodeJS-based tool called [`browser-UI-test`] that uses [puppeteer] to run tests in a headless browser and check rendering and interactivity. For information on how to write this form of test, see [`tests/rustdoc-gui/README.md`][rustdoc-gui-readme] as well as [the description of the `.goml` format][goml-script]
+
+[Rustdoc test suites]: ../tests/compiletest.md#rustdoc-test-suites
+[`browser-UI-test`]: https://github.com/GuillaumeGomez/browser-UI-test/
+[puppeteer]: https://pptr.dev/
+[rustdoc-gui-readme]: https://github.com/rust-lang/rust/blob/master/tests/rustdoc-gui/README.md
+[goml-script]: https://github.com/GuillaumeGomez/browser-UI-test/blob/master/goml-script.md

src/rustdoc-internals/rustdoc-json-test-suite.md

@@ -0,0 +1,3 @@
+# The `rustdoc-json` test suite
+
+> **FIXME**: This section is a stub. It will be populated by [PR #2422](https://github.com/rust-lang/rustc-dev-guide/pull/2422/).

src/rustdoc-internals/rustdoc-test-suite.md

@@ -1,112 +1,190 @@
 # The `rustdoc` test suite
 
-This page is specifically about the test suite named `rustdoc`.
-For other test suites used for testing rustdoc, see [Rustdoc tests](../rustdoc.md#tests).
+This page is about the test suite named `rustdoc` used to test the HTML output of `rustdoc`.
+For other rustdoc-specific test suites, see [Rustdoc test suites].
 
-The `rustdoc` test suite is specifically used to test the HTML output of rustdoc.
+Each test file in this test suite is simply a Rust source file `file.rs` sprinkled with
+so-called *directives* located inside normal Rust code comments.
+These come in two flavors: *Compiletest* and *HtmlDocCk*.
 
-This is achieved by means of `htmldocck.py`, a custom checker script that leverages [XPath].
+To learn more about the former, read [Compiletest directives].
+For the latter, continue reading.
 
-[XPath]: https://en.wikipedia.org/wiki/XPath
+Internally, [`compiletest`] invokes the supplementary checker script [`htmldocck.py`].
 
-## Directives
-Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments.
+[Rustdoc test suites]: ../tests/compiletest.md#rustdoc-test-suites
+[`compiletest`]: ../tests/compiletest.md
+[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py
 
-In addition to the directives listed here,
-`rustdoc` tests also support most
-[compiletest directives](../tests/directives.html).
+## HtmlDocCk Directives
 
-All `PATH`s in directives are relative to the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`),
-so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid
-having to write a long crate name multiple times.
-To avoid repetition, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument.
+Directives to HtmlDocCk are assertions that place constraints on the generated HTML.
+They look similar to those given to `compiletest` in that they take the form of `//@` comments
+but ultimately, they are completey distinct and processed by different programs.
 
-All arguments take the form of quoted strings
-(both single and double quotes are supported),
-with the exception of `COUNT` and the special `-` form of `PATH`.
+[XPath] is used to query parts of the HTML document tree.
+
+**Introductory example**:
+
+```rust,ignore (illustrative)
+//@ has file/type.Alias.html
+//@ has - '//*[@class="rust item-decl"]//code' 'type Alias = Option<i32>;'
+pub type Alias = Option<i32>;
+```
+
+Here, we check that documentation generated for crate `file` contains a page for the
+public type alias `Alias` where the code block that is found at the top contains the
+expected rendering of the item. The `//*[@class="rust item-decl"]//code` is an XPath
+expression.
 
-Directives are assertions that place constraints on the generated HTML.
+Conventionally, you place these directives directly above the thing they are meant to test.
+Technically speaking however, they don't need to be as HtmlDocCk only looks for the directives.
 
-All directives (except `files`) can be negated by putting a `!` in front of their name.
+All directives take a `PATH` argument.
+To avoid repetition, `-` can be passed to it to re-use the previous `PATH` argument.
+Since the path contains the name of the crate, it is conventional to add a
+`#![crate_name = "foo"]` attribute to the crate root to shorten the resulting path.
+
+All arguments take the form of shell-style (single or double) quoted strings,
+with the exception of `COUNT` and the special `-` form of `PATH`.
+
+All directives (except `files`) can be *negated* by putting a `!` in front of their name.
+Before you add negated directives, please read about [their caveats](#caveats).
 
 Similar to shell commands,
 directives can extend across multiple lines if their last char is `\`.
 In this case, the start of the next line should be `//`, with no `@`.
 
-For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root.
+Use the special string `{{channel}}` in XPaths, `PATTERN` arguments and [snapshot files](#snapshot)
+if you'd like to refer to the URL `https://doc.rust-lang.org/CHANNEL` where `CHANNEL` refers to the
+current release channel (e.g, `stable` or `nightly`).
+
+Listed below are all possible directives:
+
+[XPath]: https://en.wikipedia.org/wiki/XPath
 
 ### `has`
 
-Usage 1: `//@ has PATH`
-Usage 2: `//@ has PATH XPATH PATTERN`
+> Usage 1: `//@ has PATH`
 
-In the first form, `has` checks that a given file exists.
+Check that the file given by `PATH` exists.
 
-In the second form, `has` is an alias for `matches`,
-except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
+> Usage 2: `//@ has PATH XPATH PATTERN`
 
-### `matches`
+Checks that the text of each element / attribute / text selected by `XPATH` in the
+whitespace-normalized[^1] file given by `PATH` matches the
+(also whitespace-normalized) string `PATTERN`.
+
+**Tip**: If you'd like to avoid whitespace normalization and/or if you'd like to match with a regex,
+use `matches` instead.
 
-Usage: `//@ matches PATH XPATH PATTERN`
+### `hasraw`
 
-Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`.
+> Usage: `//@ hasraw PATH PATTERN`
 
-### `matchesraw`
+Checks that the contents of the whitespace-normalized[^1] file given by `PATH`
+matches the (also whitespace-normalized) string `PATTERN`.
 
-Usage: `//@ matchesraw PATH PATTERN`
+**Tip**: If you'd like to avoid whitespace normalization and / or if you'd like to match with a
+regex, use `matchesraw` instead.
 
-Checks that the contents of the file `PATH` matches the regex `PATTERN`.
+### `matches`
 
-### `hasraw`
+> Usage: `//@ matches PATH XPATH PATTERN`
 
-Usage: `//@ hasraw PATH PATTERN`
+Checks that the text of each element / attribute / text selected by `XPATH` in the
+file given by `PATH` matches the Python-flavored[^2] regex `PATTERN`.
 
-Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
+### `matchesraw`
+
+> Usage: `//@ matchesraw PATH PATTERN`
+
+Checks that the contents of the file given by `PATH` matches the
+Python-flavored[^2] regex `PATTERN`.
 
 ### `count`
 
-Usage: `//@ count PATH XPATH COUNT`
+> Usage: `//@ count PATH XPATH COUNT`
 
-Checks that there are exactly `COUNT` matches for `XPATH` within the file `PATH`.
+Checks that there are exactly `COUNT` matches for `XPATH` within the file given by `PATH`.
 
 ### `snapshot`
 
-Usage: `//@ snapshot NAME PATH XPATH`
+> Usage: `//@ snapshot NAME PATH XPATH`
 
-Creates a snapshot test named NAME.
-A snapshot test captures a subtree of the DOM, at the location
-determined by the XPath, and compares it to a pre-recorded value
-in a file. The file's name is the test's name with the `.rs` extension
-replaced with `.NAME.html`, where NAME is the snapshot's name.
+Checks that the element / text selected by `XPATH` in the file given by `PATH` matches the
+pre-recorded subtree or text (the "snapshot") in file `FILE_STEM.NAME.html` where `FILE_STEM`
+is the file stem of the test file.
 
-htmldocck supports the `--bless` option to accept the current subtree
-as expected, saving it to the file determined by the snapshot's name.
-compiletest's `--bless` flag is forwarded to htmldocck.
+Pass the `--bless` option to `compiletest` to accept the current subtree/text as expected.
+This will overwrite the aforementioned file (or create it if it doesn't exist). It will
+automatically normalize the channel-dependent URL `https://doc.rust-lang.org/CHANNEL` to
+the special string `{{channel}}`.
 
 ### `has-dir`
 
-Usage: `//@ has-dir PATH`
+> Usage: `//@ has-dir PATH`
 
-Checks for the existence of directory `PATH`.
+Checks for the existence of the directory given by `PATH`.
 
 ### `files`
 
-Usage: `//@ files PATH ENTRIES`
+> Usage: `//@ files PATH ENTRIES`
+
+Checks that the directory given by `PATH` contains exactly `ENTRIES`.
+`ENTRIES` is a Python-like list of strings inside a quoted string.
+
+**Example**: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'`
+
+[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space.
+[^2]: They are Unicode aware (flag `UNICODE` is set), match case-sensitively and in single-line mode.
+
+## Compiletest Directives (Brief)
+
+As mentioned in the introduction, you also have access to [compiletest directives].
+Most importantly, they allow you to register auxiliary crates and
+to pass flags to the `rustdoc` binary under test.
+It's *strongly recommended* to read that chapter if you don't know anything about them yet.
+
+Here are some details that are relevant to this test suite specifically:
 
-Checks that the directory `PATH` contains exactly `ENTRIES`.
-`ENTRIES` is a python list of strings inside a quoted string,
-as if it were to be parsed by `eval`.
-(note that the list is actually parsed by `shlex.split`,
-so it cannot contain arbitrary python expressions).
+* While you can use both `//@ compile-flags` and `//@ doc-flags` to pass flags to `rustdoc`,
+  prefer to user the latter to show intent. The former is meant for `rustc`.
+* Add `//@ build-aux-docs` to the test file that has auxiliary crates to not only compile the
+  auxiliaries with `rustc` but to also document them with `rustdoc`
 
-Example: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'`
+## Caveats
 
-[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space.  The files themselves are also whitespace-normalized.
+Testing for the absence of an element or a piece of text is quite fragile and not very future proof.
+
+It's not unusual that the *shape* of the generated HTML document tree changes from time to time.
+This includes for example renamings of CSS classes.
+
+Whenever that happens, *positive* checks will either continue to match the intended element /
+attribute / text if their XPath expression is general / loose enough and thus test the correct thing
+or they won't in which case they would fail forcing the author of the change tolook at them.
+
+Compare that to *negative* checks (e.g., `//@ !has PATH XPATH PATTERN`) which won't fail if their
+XPath expression "no longer" matches. The author who changed "the shape" thus won't get notified and
+as a result someone else can unintentionally reintroduce `PATTERN` into the generated docs without
+the original negative check failing.
+
+**Note**: Please avoid the use of *negated* checks!
+
+**Tip**: If you can't avoid it, please **always** pair it with an analogous positive check in the
+immediate vicinity, so people changing "the shape" have a chance to notice and to update the
+negated check!
 
 ## Limitations
-`htmldocck.py` uses the xpath implementation from the standard library.
+
+HtmlDocCk uses the XPath implementation from the Python standard library.
 This leads to several limitations:
+
 * All `XPATH` arguments must start with `//` due to a flaw in the implementation.
 * Many XPath features (functions, axies, etc.) are not supported.
 * Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags).
 
+Furthmore, compiletest [revisions] are not supported.
+
+[revisions]: ../tests/compiletest.md#revisions
+[compiletest directives]: ../tests/directives.md

src/rustdoc.md

@@ -67,43 +67,29 @@ does is call the `main()` that's in this crate's `lib.rs`, though.)
 
 ## Code structure
 
-* All paths in this section are relative to `src/librustdoc` in the rust-lang/rust repository.
+All paths in this section are relative to `src/librustdoc/` in the rust-lang/rust repository.
+
 * Most of the HTML printing code is in `html/format.rs` and `html/render/mod.rs`.
-  It's in a bunch of `fmt::Display` implementations and supplementary
-  functions.
-* The types that got `Display` impls above are defined in `clean/mod.rs`, right
-  next to the custom `Clean` trait used to process them out of the rustc HIR.
+  It's in a bunch of functions returning `impl std::fmt::Display`.
+* The data types that get rendered by the functions mentioned above are defined in `clean/types.rs`.
+  The functions responsible for creating them from the `HIR` and the `rustc_middle::ty` IR
+  live in `clean/mod.rs`.
 * The bits specific to using rustdoc as a test harness are in
   `doctest.rs`.
 * The Markdown renderer is loaded up in `html/markdown.rs`, including functions
   for extracting doctests from a given block of Markdown.
 * Frontend CSS and JavaScript are stored in `html/static/`.
+  * Re. JavaScript, type annotations are written using [TypeScript-flavored JSDoc]
+comments and an external `.d.ts` file.
+    This way, the code itself remains plain, valid JavaScript.
+    We only use `tsc` as a linter.
 
-## Tests
+[TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html
 
-* Tests on search engine and index are located in `tests/rustdoc-js` and `tests/rustdoc-js-std`.
-  The format is specified
-  [in the search guide](rustdoc-internals/search.md#testing-the-search-engine).
-* Tests on the "UI" of rustdoc (the terminal output it produces when run) are in
-  `tests/rustdoc-ui`
-* Tests on the "GUI" of rustdoc (the HTML, JS, and CSS as rendered in a browser)
-  are in `tests/rustdoc-gui`. These use a [NodeJS tool called
-  browser-UI-test](https://github.com/GuillaumeGomez/browser-UI-test/) that uses
-  puppeteer to run tests in a headless browser and check rendering and
-  interactivity.  For information on how to write this form of test,
-  see [`tests/rustdoc-gui/README.md`][rustdoc-gui-readme]
-  as well as [the description of the `.goml` format][goml-script]
-* Tests on the structure of rustdoc HTML output are located in `tests/rustdoc`,
-  where they're handled by the test runner of bootstrap and
-  the supplementary script `src/etc/htmldocck.py`.
-  [These tests have several extra directives available to them](./rustdoc-internals/rustdoc-test-suite.md).
-* Additionally, JavaScript type annotations are written using [TypeScript-flavored JSDoc]
-  comments and an external d.ts file. The code itself is plain, valid JavaScript; we only
-  use tsc as a linter.
+## Tests
 
-[TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html
-[rustdoc-gui-readme]: https://github.com/rust-lang/rust/blob/master/tests/rustdoc-gui/README.md
-[goml-script]: https://github.com/GuillaumeGomez/browser-UI-test/blob/master/goml-script.md
+`rustdoc`'s integration tests are split across several test suites.
+See [Rustdoc tests suites](tests/compiletest.md#rustdoc-test-suites) for more details.
 
 ## Constraints