|
| 1 | +# `compiletest` |
| 2 | +## Introduction |
| 3 | +`compiletest` is the main test harness of the Rust test suite. It allows test authors to organize large numbers of tests (the |
| 4 | +Rust compiler has many thousands), efficient test execution (parallel execution is supported), and allows the test author to |
| 5 | +configure behavior and expected results of both individual and groups of tests. |
| 6 | + |
| 7 | +`compiletest` tests may check test code for success, for failure or in some cases, even failure to compile. Tests are |
| 8 | +typically organized as a Rust source file with annotations in comments before and/or within the test code, which serve to |
| 9 | +direct `compiletest` on if or how to run the test, what behavior to expect, and more. If you are unfamiliar with the compiler |
| 10 | +testing framework, see [`this chapter`](./tests/intro.html) for additional background. |
| 11 | + |
| 12 | +The tests themselves are typically (but not always) organized into "suites"--for example, `run-pass`, a folder |
| 13 | +representing tests that should succeed, `run-fail`, a folder holding tests that should compile successfully, but return |
| 14 | +a failure (non-zero status), `compile-fail`, a folder holding tests that should fail to compile, and many more. The various |
| 15 | +suites are defined in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs) in the `pub struct Config` declaration. And a very good |
| 16 | +introduction to the different suites of compiler tests along with details about them can be found in [`Adding new tests`](./tests/adding.html). |
| 17 | + |
| 18 | +## Adding a new test file |
| 19 | +Briefly, simply create your new test in the appropriate location under [src/test](https://github.com/rust-lang/rust/tree/master/src/test). No registration of test files is necessary as |
| 20 | +`compiletest` will scan the [src/test](https://github.com/rust-lang/rust/tree/master/src/test) subfolder recursively, and will execute any Rust source files it finds as tests. |
| 21 | +See [`Adding new tests`](./tests/adding.html) for a complete guide on how to adding new tests. |
| 22 | + |
| 23 | +## Header Commands |
| 24 | +Source file annotations which appear in comments near the top of the source file *before* any test code are known as header |
| 25 | +commands. These commands can instruct `compiletest` to ignore this test, set expectations on whether it is expected to |
| 26 | +succeed at compiling, or what the test's return code is expected to be. Header commands (and their inline counterparts, |
| 27 | +Error Info commands) are described more fully [here](./tests/adding.html#header-commands-configuring-rustc). |
| 28 | + |
| 29 | +### Adding a new header command |
| 30 | +Header commands are defined in the `TestProps` struct in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs). At a high level, there are dozens of test properties defined here, all set to default values in the `TestProp` struct's `impl` block. Any test can override this |
| 31 | +default value by specifying the property in question as header command as a comment (`//`) in the test source file, before any source code. |
| 32 | + |
| 33 | +#### Using a header command |
| 34 | +Here is an example, specifying the `must-compile-successfully` header command, which takes no arguments, followed by the |
| 35 | +`failure-status` header command, which takes a single argument (which, in this case is a value of 1). `failure-status` is |
| 36 | +instructing `compiletest` to expect a failure status of 1 (rather than the current Rust default of 101 at the time of this |
| 37 | +writing). The header command and the argument list (if present) are typically separated by a colon: |
| 38 | +``` |
| 39 | +// Copyright 2018 The Rust Project Developers. See the COPYRIGHT |
| 40 | +// file at the top-level directory of this distribution and at |
| 41 | +// http://rust-lang.org/COPYRIGHT. |
| 42 | +// |
| 43 | +// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or |
| 44 | +// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license |
| 45 | +// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your |
| 46 | +// option. This file may not be copied, modified, or distributed |
| 47 | +// except according to those terms. |
| 48 | +
|
| 49 | +// must-compile-successfully |
| 50 | +// failure-status: 1 |
| 51 | +
|
| 52 | +#![feature(termination_trait)] |
| 53 | +
|
| 54 | +use std::io::{Error, ErrorKind}; |
| 55 | +
|
| 56 | +fn main() -> Result<(), Box<Error>> { |
| 57 | + Err(Box::new(Error::new(ErrorKind::Other, "returned Box<Error> from main()"))) |
| 58 | +} |
| 59 | +``` |
| 60 | + |
| 61 | +#### Adding a new header command property |
| 62 | +One would add a new header command if there is a need to define some test property or behavior on an individual, test-by-test |
| 63 | +basis. A header command property serves as the header command's backing store (holds the command's current value) at |
| 64 | +runtime. |
| 65 | + |
| 66 | +To add a new header command property: |
| 67 | + 1. Look for the `pub struct TestProps` declaration in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs) and add |
| 68 | +the new public property to the end of the declaration. |
| 69 | + 2. Look for the `impl TestProps` implementation block immediately following the struct declaration and initialize the new |
| 70 | +property to its default value. |
| 71 | + |
| 72 | +#### Adding a new header command parser |
| 73 | +When `compiletest` encounters a test file, it parses the file a line at a time by calling every parser defined in the |
| 74 | +`Config` struct's implementation block, also in [src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs) (note the `Config` struct's declaration |
| 75 | +block is found in [src/tools/compiletest/src/common.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs). `TestProps`'s `load_from()` method will try passing the current |
| 76 | +line of text to each parser, which, in turn typically checks to see if the line begins with a particular commented (`//`) |
| 77 | +header command such as `// must-compile-successfully` or `// failure-status`. Whitespace after the comment marker is |
| 78 | +optional. |
| 79 | + |
| 80 | +Parsers will override a given header command property's default value merely by being specified in the test file as a header |
| 81 | +command or by having a parameter value specified in the test file, depending on the header command. |
| 82 | + |
| 83 | +Parsers defined in `impl Config` are typically named `parse_<header_command>` (note kebab-case `<header-command>` transformed |
| 84 | +to snake-case `<header_command>`). `impl Config` also defines several 'low-level' parsers which make it simple to parse |
| 85 | +common patterns like simple presence or not (`parse_name_directive()`), header-command:parameter(s) |
| 86 | +(`parse_name_value_directive()`), optional parsing only if a particular `cfg` attribute is defined (`has_cfg_prefix()`) and |
| 87 | +many more. The low-level parsers are found near the end of the `impl Config` block; be sure to look through them and their |
| 88 | +associated parsers immediately above to see how they are used to avoid writing additional parsing code unneccessarily. |
| 89 | + |
| 90 | +As a concrete example, here is the implementation for the `parse_failure_status()` parser, in |
| 91 | +[src/tools/compiletest/src/header.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs): |
| 92 | +```diff |
| 93 | +@@ -232,6 +232,7 @@ pub struct TestProps { |
| 94 | + // customized normalization rules |
| 95 | + pub normalize_stdout: Vec<(String, String)>, |
| 96 | + pub normalize_stderr: Vec<(String, String)>, |
| 97 | ++ pub failure_status: i32, |
| 98 | + } |
| 99 | + |
| 100 | + impl TestProps { |
| 101 | +@@ -260,6 +261,7 @@ impl TestProps { |
| 102 | + run_pass: false, |
| 103 | + normalize_stdout: vec![], |
| 104 | + normalize_stderr: vec![], |
| 105 | ++ failure_status: 101, |
| 106 | + } |
| 107 | + } |
| 108 | + |
| 109 | +@@ -383,6 +385,10 @@ impl TestProps { |
| 110 | + if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") { |
| 111 | + self.normalize_stderr.push(rule); |
| 112 | + } |
| 113 | ++ |
| 114 | ++ if let Some(code) = config.parse_failure_status(ln) { |
| 115 | ++ self.failure_status = code; |
| 116 | ++ } |
| 117 | + }); |
| 118 | + |
| 119 | + for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] { |
| 120 | +@@ -488,6 +494,13 @@ impl Config { |
| 121 | + self.parse_name_directive(line, "pretty-compare-only") |
| 122 | + } |
| 123 | + |
| 124 | ++ fn parse_failure_status(&self, line: &str) -> Option<i32> { |
| 125 | ++ match self.parse_name_value_directive(line, "failure-status") { |
| 126 | ++ Some(code) => code.trim().parse::<i32>().ok(), |
| 127 | ++ _ => None, |
| 128 | ++ } |
| 129 | ++ } |
| 130 | +``` |
| 131 | + |
| 132 | +## Implementing the behavior change |
| 133 | +When a test invokes a particular header command, it is expected that some behavior will change as a result. What behavior, |
| 134 | +obviously, will depend on the purpose of the header command. In the case of `failure-status`, the behavior that changes |
| 135 | +is that `compiletest` expects the failure code defined by the header command invoked in the test, rather than the default |
| 136 | +value. |
| 137 | + |
| 138 | +Although specific to `failure-status` (as every header command will have a different implementation in order to invoke |
| 139 | +behavior change) perhaps it is helpful to see the behavior change implementation of one case, simply as an example. To implement `failure-status`, the `check_correct_failure_status()` function found in the `TestCx` implementation block, |
| 140 | +located in [src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs), was modified as per below: |
| 141 | +```diff |
| 142 | +@@ -295,11 +295,14 @@ impl<'test> TestCx<'test> { |
| 143 | + } |
| 144 | + |
| 145 | + fn check_correct_failure_status(&self, proc_res: &ProcRes) { |
| 146 | +- // The value the rust runtime returns on failure |
| 147 | +- const RUST_ERR: i32 = 101; |
| 148 | +- if proc_res.status.code() != Some(RUST_ERR) { |
| 149 | ++ let expected_status = Some(self.props.failure_status); |
| 150 | ++ let received_status = proc_res.status.code(); |
| 151 | ++ |
| 152 | ++ if expected_status != received_status { |
| 153 | + self.fatal_proc_rec( |
| 154 | +- &format!("failure produced the wrong error: {}", proc_res.status), |
| 155 | ++ &format!("Error: expected failure status ({:?}) but received status {:?}.", |
| 156 | ++ expected_status, |
| 157 | ++ received_status), |
| 158 | + proc_res, |
| 159 | + ); |
| 160 | + } |
| 161 | +@@ -320,7 +323,6 @@ impl<'test> TestCx<'test> { |
| 162 | + ); |
| 163 | + |
| 164 | + let proc_res = self.exec_compiled_test(); |
| 165 | +- |
| 166 | + if !proc_res.status.success() { |
| 167 | + self.fatal_proc_rec("test run failed!", &proc_res); |
| 168 | + } |
| 169 | +@@ -499,7 +501,6 @@ impl<'test> TestCx<'test> { |
| 170 | + expected, |
| 171 | + actual |
| 172 | + ); |
| 173 | +- panic!(); |
| 174 | + } |
| 175 | + } |
| 176 | +``` |
| 177 | +Note the use of `self.props.failure_status` to access the header command property. In tests which do not specify the failure |
| 178 | +status header command, `self.props.failure_status` will evaluate to the default value of 101 at the time of this writing. |
| 179 | +But for a test which specifies a header command of, for example, `// failure-status: 1`, `self.props.failure_status` will |
| 180 | +evaluate to 1, as `parse_failure_status()` will have overridden the `TestProps` default value, for that test specifically. |
0 commit comments