add new section on the `rustdoc` test suite #2295

lolbinarycat · 2025-03-18T17:08:17Z

Currently this stuff is only documented in a comment at the start of htmldocck.py, and that commend is very outdated and often missing important details, like the fact that has and matches are accept different kinds of patterns.

fmease

Thanks for working on this. I have a couple of comments!

src/tests/compiletest.md

src/rustdoc-internals/htmldocck.md

fmease · 2025-03-18T17:57:39Z

Also CC @GuillaumeGomez @notriddle

src/rustdoc-internals/htmldocck.md

lolbinarycat · 2025-03-18T18:28:01Z

@fmease I think I've managed to address all your comments in a way I'm satisfied with, does this look good?

jieyouxu · 2025-03-20T06:49:00Z

Merging since rustdoc maintainers approved, can always be improved in follow-ups.

jieyouxu · 2025-03-20T06:49:45Z

(rust-lang/team#1704 will allow rustdoc maintainers to have write access to this repo, since it makes no sense if the rustdoc experts can't merge rustdoc doc changes.)

fmease · 2025-03-20T07:08:30Z

Noo, my review wasn't done (this still has several smallish issues) :/ I'll create a follow-up PR

jieyouxu · 2025-03-20T07:14:32Z

Oh sorry :(

fmease · 2025-03-19T22:20:42Z

src/rustdoc-internals/rustdoc-test-suite.md

+
+The `rustdoc` test suite is specifically used to test the HTML output of rustdoc.
+
+This is achived by means of `htmldocck.py`, a custom checker script that leverages [XPath].


Suggested change

This is achived by means of `htmldocck.py`, a custom checker script that leverages [XPath].

This is achieved by means of `htmldocck.py`, a custom checker script that leverages [XPath].

fmease · 2025-03-19T22:24:41Z

src/rustdoc-internals/rustdoc-test-suite.md

+
+Usage: `//@ matches PATH XPATH PATTERN`
+
+Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`.


Suggested change

Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`.

Checks that the text of each element selected by `XPATH` in `PATH` matches the Python-flavored regex `PATTERN`.

fmease · 2025-03-19T22:27:00Z

src/rustdoc-internals/rustdoc-test-suite.md

+
+In the first form, `has` checks that a given file exists.
+
+In the second form, `has` is an alias for `matches`,


Suggested change

In the second form, `has` is an alias for `matches`,

In the second form, `has` is the same as `matches`,

If it's not the same it's not an alias :)

fmease · 2025-03-19T22:33:09Z

src/rustdoc-internals/rustdoc-test-suite.md

+
+Usage: `//@ has-dir PATH`
+
+Checks for the existance of directory `PATH`.


Suggested change

Checks for the existance of directory `PATH`.

Checks for the existence of directory `PATH`.

fmease · 2025-03-19T22:34:25Z

src/rustdoc-internals/rustdoc-test-suite.md

+[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space.  The files themselves are also whitespace-normalized.
+
+## Limitations
+`htmldocck.py` uses the xpath implementation from the standard library.


Suggested change

`htmldocck.py` uses the xpath implementation from the standard library.

`htmldocck.py` uses the XPath implementation from the standard library.

fmease · 2025-03-19T22:36:02Z

src/rustdoc-internals/rustdoc-test-suite.md

+`ENTRIES` is a python list of strings inside a quoted string,
+as if it were to be parsed by `eval`.
+(note that the list is actually parsed by `shlex.split`,
+so it cannot contain arbitrary python expressions).


Suggested change

`ENTRIES` is a python list of strings inside a quoted string,

as if it were to be parsed by `eval`.

(note that the list is actually parsed by `shlex.split`,

so it cannot contain arbitrary python expressions).

`ENTRIES` is a Python list of strings inside a quoted string,

as if it were to be parsed by `eval`.

(note that the list is actually parsed by `shlex.split`,

so it cannot contain arbitrary Python expressions).

fmease · 2025-03-19T23:08:11Z

src/rustdoc-internals/rustdoc-test-suite.md

+
+### `hasraw`
+
+Usage: `//@ hasraw PATH PATTERN`


Suggested change

Usage: `//@ hasraw PATH PATTERN`

Usage: `//@ hasraw PATH XPATH PATTERN`

+ extending prose description

fmease · 2025-03-19T23:08:19Z

src/rustdoc-internals/rustdoc-test-suite.md

+
+### `matchesraw`
+
+Usage: `//@ matchesraw PATH PATTERN`


Suggested change

Usage: `//@ matchesraw PATH PATTERN`

Usage: `//@ matchesraw PATH XPATH PATTERN`

+ extending prose description

fmease · 2025-03-20T07:12:38Z

src/rustdoc-internals/rustdoc-test-suite.md

+
+Usage: `//@ hasraw PATH PATTERN`
+
+Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.


This only mentions in the footnote that file's contents are also whitespace normalized. However, that's arguably more important (in 99% cases the PATTERN the test author writes is normalized already). This should be the other way around.

fmease · 2025-03-20T07:14:49Z

src/rustdoc-internals/rustdoc-test-suite.md

+In the first form, `has` checks that a given file exists.
+
+In the second form, `has` is an alias for `matches`,
+except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.


fmease · 2025-03-20T07:15:32Z

Also this should've been squashed ^^' (the last 7 commits were just fixups and thus should't've become part of the final history)

jieyouxu · 2025-03-20T07:29:49Z

Noo, my review wasn't done (this still has several smallish issues) :/ I'll create a follow-up PR

I can address some of the stuff you mentioned unless you already have changes locally, I'll also add a bit more on how this is wired up between bootstrap <-> rustdoc test suites.

(Sorry)

fmease · 2025-03-20T07:31:46Z

src/rustdoc-internals/rustdoc-test-suite.md

https://rustc-dev-guide.rust-lang.org/rustdoc-internals.html#dotting-is-and-crossing-ts already mentions htmldocck albeit more briefly. Furthermore it mentions useful compiletest directives like //@ build-aux-docs.

We should remove this section and import some of the additional pieces of information into this document. We should also mention compiletest directive //@ doc-flags.

fmease · 2025-03-20T07:33:56Z

I already have my editor open and will open a PR in <30 minutes I guess. Feel free to add extra information after that :)

fmease · 2025-03-20T07:38:20Z

src/rustdoc-internals/rustdoc-test-suite.md

+`rustdoc` tests also support most
+[compiletest directives](../tests/directives.html).
+
+All `PATH`s in directives are relative to the the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`),


Unless I'm confusing myself, if you say the path is relative to build/$TARGET/test/rustdoc/$CRATE_NAME I would think a path like foo/fn.bar.html would expand to build/$TARGET/test/rustdoc/foo/foo/fn.bar.html assuming the crate name is foo (notice the double foo/).

I would describe it as relative to build/$TARGET/test/rustdoc/ in which case this sentence merely describes an implementation detail of compiletest which the test writer shouldn't need to worry about.

Is tests/rustdoc supposed to support revisions? I'm assuming the answer is "no", but I don't see it being explicitly rejected.

(Asking because revision is part of the output dir name mangling)

They're currently rejected by compiletest (via an assert) since htmldocck.py has no notion of that

Right, https://github.com/rust-lang/rust/blob/4e2b096ed6c8a1400624a54f6c4fd0c0ce48a579/src/tools/compiletest/src/runtest/rustdoc.rs#L7. I was looking at another place where I did the rejection 💀 I should fix that as part of the compiletest directive handling rework.

lolbinarycat added 2 commits March 18, 2025 12:04

add new section on the rustdoc test suite

f00643a

add htmldocck.md to SUMMARY.md

8b50156

fmease self-assigned this Mar 18, 2025

fmease added S-waiting-on-review Status: this PR is waiting for a reviewer to verify its content T-rustdoc Relevant to rustdoc team labels Mar 18, 2025

fmease reviewed Mar 18, 2025

View reviewed changes

htmldocck: expand limitations and mention compiletest directives

f248d2f

fmease reviewed Mar 18, 2025

View reviewed changes

src/rustdoc-internals/htmldocck.md Outdated Show resolved Hide resolved

lolbinarycat added 4 commits March 18, 2025 13:19

rustdoc test suite: clean up wording and intro

72aa06d

rename htmldocck.md -> rustdoc-test-suite.md

174678d

clean up wording/grammar and mention double quotes

2e1c499

normalize link titles

871280d

update filename in link

59f11cd

notriddle approved these changes Mar 19, 2025

View reviewed changes

jieyouxu merged commit 047ccc0 into rust-lang:master Mar 20, 2025
1 check passed

fmease reviewed Mar 20, 2025

View reviewed changes

fmease mentioned this pull request Mar 20, 2025

[WIP] rustdoc testing: Further improve chapters and sections #2298

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add new section on the `rustdoc` test suite #2295

add new section on the `rustdoc` test suite #2295

lolbinarycat commented Mar 18, 2025

fmease left a comment

fmease commented Mar 18, 2025

lolbinarycat commented Mar 18, 2025

jieyouxu commented Mar 20, 2025

jieyouxu commented Mar 20, 2025

fmease commented Mar 20, 2025 •

edited

Loading

jieyouxu commented Mar 20, 2025

fmease Mar 19, 2025

fmease Mar 19, 2025

fmease Mar 19, 2025

fmease Mar 19, 2025

fmease Mar 19, 2025

fmease Mar 19, 2025

fmease Mar 19, 2025

fmease Mar 19, 2025

fmease Mar 20, 2025 •

edited

Loading

fmease Mar 20, 2025 •

edited

Loading

fmease commented Mar 20, 2025 •

edited

Loading

jieyouxu commented Mar 20, 2025

fmease Mar 20, 2025

fmease commented Mar 20, 2025

fmease Mar 20, 2025

jieyouxu Mar 20, 2025 •

edited

Loading

fmease Mar 20, 2025

jieyouxu Mar 20, 2025


		The `rustdoc` test suite is specifically used to test the HTML output of rustdoc.

		This is achived by means of `htmldocck.py`, a custom checker script that leverages [XPath].

	This is achived by means of `htmldocck.py`, a custom checker script that leverages [XPath].
	This is achieved by means of `htmldocck.py`, a custom checker script that leverages [XPath].


		Usage: `//@ matches PATH XPATH PATTERN`

		Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`.


		In the first form, `has` checks that a given file exists.

		In the second form, `has` is an alias for `matches`,

	In the second form, `has` is an alias for `matches`,
	In the second form, `has` is the same as `matches`,


		Usage: `//@ has-dir PATH`

		Checks for the existance of directory `PATH`.

	Checks for the existance of directory `PATH`.
	Checks for the existence of directory `PATH`.

	`htmldocck.py` uses the xpath implementation from the standard library.
	`htmldocck.py` uses the XPath implementation from the standard library.

	Usage: `//@ hasraw PATH PATTERN`
	Usage: `//@ hasraw PATH XPATH PATTERN`

	Usage: `//@ matchesraw PATH PATTERN`
	Usage: `//@ matchesraw PATH XPATH PATTERN`


		Usage: `//@ hasraw PATH PATTERN`

		Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.

add new section on the rustdoc test suite #2295

add new section on the rustdoc test suite #2295

Conversation

lolbinarycat commented Mar 18, 2025

fmease left a comment

Choose a reason for hiding this comment

fmease commented Mar 18, 2025

lolbinarycat commented Mar 18, 2025

jieyouxu commented Mar 20, 2025

jieyouxu commented Mar 20, 2025

fmease commented Mar 20, 2025 • edited Loading

jieyouxu commented Mar 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmease Mar 20, 2025 • edited Loading

Choose a reason for hiding this comment

fmease Mar 20, 2025 • edited Loading

Choose a reason for hiding this comment

fmease commented Mar 20, 2025 • edited Loading

jieyouxu commented Mar 20, 2025

Choose a reason for hiding this comment

fmease commented Mar 20, 2025

Choose a reason for hiding this comment

jieyouxu Mar 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

add new section on the `rustdoc` test suite #2295

add new section on the `rustdoc` test suite #2295

fmease commented Mar 20, 2025 •

edited

Loading

fmease Mar 20, 2025 •

edited

Loading

fmease Mar 20, 2025 •

edited

Loading

fmease commented Mar 20, 2025 •

edited

Loading

jieyouxu Mar 20, 2025 •

edited

Loading