From 10dd2d72c81ee788a9829d8e44e2c9c28e9aefd2 Mon Sep 17 00:00:00 2001 From: Michael Howell Date: Fri, 23 Aug 2024 10:31:02 -0700 Subject: [PATCH 1/6] Add docs for JS tests --- src/rustdoc-internals/search.md | 99 +++++++++++++++++++++++++++++++++ src/rustdoc.md | 3 +- 2 files changed, 101 insertions(+), 1 deletion(-) diff --git a/src/rustdoc-internals/search.md b/src/rustdoc-internals/search.md index 8ade98763..cd1cbf1ba 100644 --- a/src/rustdoc-internals/search.md +++ b/src/rustdoc-internals/search.md @@ -354,3 +354,102 @@ The unification filter ensures that: The bloom filter checks none of these things, and, on top of that, can have false positives. But it's fast and uses very little memory, so the bloom filter helps. + +## Testing the search engine + +While the DOM UI code itself is tested using `rustdoc-gui` tests, the +primary way the search engine is tested is the `rustdoc-js` and +`rustdoc-js-std` tests. They run in NodeJS. + +A `rustdoc-js` test has a `.rs` and `.js` file, with the same name. +The `.rs` file specifies the hypothetical library crate to run +the searches on (make sure you mark anything you need to find as `pub`). +The `.js` file specifies the actual searches. +The `rustdoc-js-std` tests are the same, but don't require an `.rs` +file, since they use the standard library. + +The `.js` file is like a module (except the loader takes care of +`exports` for you). It expects you to set these variables in the +module's scope: + +| Name | Type | Description +| -------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------- +| `FILTER_CRATE` | `string` | Only include results from the given crate. In the GUI, this is the "Results in crate" drop-down menu. +| `EXPECTED` | `[ResultsTable]\|ResultsTable` | List of tests to run, specifying what the hypothetical user types into the search box and sees in the tabs +| `PARSED` | `[ParsedQuery]\|ParsedQuery` | List of parser tests to run, without running an actual search + +Additionally, the following magic comments are supported. +Put them on their own line, without indenting. + +* `// exact-check`: If search results appear that aren't part of the test case, + then fail. By default, the test case will tolerate the engine returning more + results than specified. +* `// ignore-order`: By default, the entries in the test case must have a + matching order in the results. Setting this option allows you to test + filtering without also testing ranking, if you expect a test to be sensitive + to minor scoring changes. +* `// should-fail`: Used to write negative tests. + +Standard library tests probably shouldn't specify any of these (we want the +libs team to be able to add new stuff without causing our tests to fail), but +standalone tests will often want `// exact-check`. + +`FILTER_CRATE` can be left out (equivalent to searching "all crates"), but you +have to specify `EXPECTED` or `PARSED`. + +The `ResultsTable` and `ParsedQuery` types are specified in +[`externs.js`](https://github.com/rust-lang/rust/blob/master/src/librustdoc/html/static/js/externs.js). + +For example, imagine we needed to fix a bug where a function named +`constructor` couldn't be found. To do this, write two files: + +```rust +// tests/rustdoc-js/constructor_search.rs +// The test case needs to find this result. +pub fn constructor(_input: &str) -> i32 { 1 } +``` + +```js +// tests/rustdoc-js/constructor_search.js +// exact-check +// Since this test runs against its own crate, +// new items should not appear in the search results. +const EXPECTED = [ + // This first test targets name-based search. + { + query: "constructor", + others: [ + { path: "constructor_search", name: "constructor" }, + ], + in_args: [], + returned: [], + }, + // This test targets the second tab. + { + query: "str", + others: [], + in_args: [ + { path: "constructor_search", name: "constructor" }, + ], + returned: [], + }, + // This test targets the third tab. + { + query: "i32", + others: [], + in_args: [], + returned: [ + { path: "constructor_search", name: "constructor" }, + ], + }, + // This test targets advanced type-driven search. + { + query: "str -> i32", + others: [ + { path: "constructor_search", name: "constructor" }, + ], + in_args: [], + returned: [], + }, +] +``` diff --git a/src/rustdoc.md b/src/rustdoc.md index dbaf0c0bf..1357ffdc5 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -83,7 +83,8 @@ does is call the `main()` that's in this crate's `lib.rs`, though.) * All paths in this section are relative to `tests` in the rust-lang/rust repository. * Tests on search index generation are located in `rustdoc-js`, as a series of JavaScript files that encode queries on the standard library search - index and expected results. + index and expected results. The format is specified + [in the search guide](rustdoc-internals/search.md#testing-the-search-engine). * Tests on the "UI" of rustdoc (the terminal output it produces when run) are in `rustdoc-ui` * Tests on the "GUI" of rustdoc (the HTML, JS, and CSS as rendered in a browser) From fa1dde57f97842ba3c3030a43b134f14480ab540 Mon Sep 17 00:00:00 2001 From: Michael Howell Date: Fri, 23 Aug 2024 10:59:11 -0700 Subject: [PATCH 2/6] Minor wording improvements --- src/rustdoc-internals/search.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/src/rustdoc-internals/search.md b/src/rustdoc-internals/search.md index cd1cbf1ba..148dbd632 100644 --- a/src/rustdoc-internals/search.md +++ b/src/rustdoc-internals/search.md @@ -369,8 +369,7 @@ The `rustdoc-js-std` tests are the same, but don't require an `.rs` file, since they use the standard library. The `.js` file is like a module (except the loader takes care of -`exports` for you). It expects you to set these variables in the -module's scope: +`exports` for you). It uses these variables: | Name | Type | Description | -------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------- @@ -378,6 +377,9 @@ module's scope: | `EXPECTED` | `[ResultsTable]\|ResultsTable` | List of tests to run, specifying what the hypothetical user types into the search box and sees in the tabs | `PARSED` | `[ParsedQuery]\|ParsedQuery` | List of parser tests to run, without running an actual search +`FILTER_CRATE` can be left out (equivalent to searching "all crates"), but you +have to specify `EXPECTED` or `PARSED`. + Additionally, the following magic comments are supported. Put them on their own line, without indenting. @@ -390,12 +392,9 @@ Put them on their own line, without indenting. to minor scoring changes. * `// should-fail`: Used to write negative tests. -Standard library tests probably shouldn't specify any of these (we want the -libs team to be able to add new stuff without causing our tests to fail), but -standalone tests will often want `// exact-check`. - -`FILTER_CRATE` can be left out (equivalent to searching "all crates"), but you -have to specify `EXPECTED` or `PARSED`. +Standard library tests usually shouldn't specify `// exact-check`, since we +want the libs team to be able to add new items without causing unrelated +tests to fail, but standalone tests will use it more often. The `ResultsTable` and `ParsedQuery` types are specified in [`externs.js`](https://github.com/rust-lang/rust/blob/master/src/librustdoc/html/static/js/externs.js). From 48bd158897448142a1f0820a61634031896a2fde Mon Sep 17 00:00:00 2001 From: Michael Howell Date: Fri, 23 Aug 2024 11:33:17 -0700 Subject: [PATCH 3/6] Clean up misleading language --- src/rustdoc.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index 1357ffdc5..b09ae319d 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -81,9 +81,8 @@ does is call the `main()` that's in this crate's `lib.rs`, though.) ## Tests * All paths in this section are relative to `tests` in the rust-lang/rust repository. -* Tests on search index generation are located in `rustdoc-js`, as a - series of JavaScript files that encode queries on the standard library search - index and expected results. The format is specified +* Tests on search engine and index are located in `rustdoc-js` and `rustdoc-js-std`. + The format is specified [in the search guide](rustdoc-internals/search.md#testing-the-search-engine). * Tests on the "UI" of rustdoc (the terminal output it produces when run) are in `rustdoc-ui` From 43aa38656f8629308ce0a0104229a93162695c01 Mon Sep 17 00:00:00 2001 From: Michael Howell Date: Fri, 23 Aug 2024 11:33:31 -0700 Subject: [PATCH 4/6] Update src/rustdoc-internals/search.md Co-authored-by: Guillaume Gomez --- src/rustdoc-internals/search.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rustdoc-internals/search.md b/src/rustdoc-internals/search.md index 148dbd632..133ded9dc 100644 --- a/src/rustdoc-internals/search.md +++ b/src/rustdoc-internals/search.md @@ -357,7 +357,7 @@ But it's fast and uses very little memory, so the bloom filter helps. ## Testing the search engine -While the DOM UI code itself is tested using `rustdoc-gui` tests, the +While the generated UI is tested using `rustdoc-gui` tests, the primary way the search engine is tested is the `rustdoc-js` and `rustdoc-js-std` tests. They run in NodeJS. From 0a8b5c24afef54e2a0832dc9fc8b7a9b8f740402 Mon Sep 17 00:00:00 2001 From: Michael Howell Date: Fri, 23 Aug 2024 13:01:06 -0700 Subject: [PATCH 5/6] Specify, more explicitly, the default behavior --- src/rustdoc-internals/search.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/src/rustdoc-internals/search.md b/src/rustdoc-internals/search.md index 133ded9dc..41f2ee0b1 100644 --- a/src/rustdoc-internals/search.md +++ b/src/rustdoc-internals/search.md @@ -380,16 +380,15 @@ The `.js` file is like a module (except the loader takes care of `FILTER_CRATE` can be left out (equivalent to searching "all crates"), but you have to specify `EXPECTED` or `PARSED`. -Additionally, the following magic comments are supported. +By default, the test fails if any of the expected results are missing, +or if the results don't appear in the specified order. +The actual search results may, however, include results that aren't in the test. +To override this, specify any of the following magic comments. Put them on their own line, without indenting. * `// exact-check`: If search results appear that aren't part of the test case, - then fail. By default, the test case will tolerate the engine returning more - results than specified. -* `// ignore-order`: By default, the entries in the test case must have a - matching order in the results. Setting this option allows you to test - filtering without also testing ranking, if you expect a test to be sensitive - to minor scoring changes. + then fail. +* `// ignore-order`: Allow search results to appear in any order. * `// should-fail`: Used to write negative tests. Standard library tests usually shouldn't specify `// exact-check`, since we From b56cf2951421ae44e7b81597cc649b5aadb5a2de Mon Sep 17 00:00:00 2001 From: Michael Howell Date: Fri, 23 Aug 2024 17:22:49 -0700 Subject: [PATCH 6/6] Further clarification --- src/rustdoc-internals/search.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/rustdoc-internals/search.md b/src/rustdoc-internals/search.md index 41f2ee0b1..dc7be250e 100644 --- a/src/rustdoc-internals/search.md +++ b/src/rustdoc-internals/search.md @@ -380,8 +380,11 @@ The `.js` file is like a module (except the loader takes care of `FILTER_CRATE` can be left out (equivalent to searching "all crates"), but you have to specify `EXPECTED` or `PARSED`. -By default, the test fails if any of the expected results are missing, -or if the results don't appear in the specified order. + + +By default, the test fails if any of the results specified in the test case are +not found after running the search, or if the results found after running the +search don't appear in the same order that they do in the test. The actual search results may, however, include results that aren't in the test. To override this, specify any of the following magic comments. Put them on their own line, without indenting.