Skip to content

Commit a24ab13

Browse files
committed
Add some docs on hints on dealing with varying tests.
1 parent 67bc835 commit a24ab13

File tree

1 file changed

+23
-1
lines changed

1 file changed

+23
-1
lines changed

Diff for: TUTORIAL.md

+23-1
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,29 @@ cargo-bisect-rustc --script=./test.sh --test-dir=foo \
233233
--end=2c2e2c57dc2140cfb62a8abb9312b89f02c59f3c
234234
```
235235

236-
236+
## Varying tests
237+
238+
When writing your test and picking a bisection range, you should be careful to
239+
ensure that the test won't vary between pass/fail over time. It should only
240+
transition from good to bad once in the bisection range (it must change
241+
[monotonically]). The following are some suggestions for dealing with a
242+
potentially varying test:
243+
244+
* Use the `-vv` flag (very verbose) to display the output from the compiler to
245+
make sure it is what you expect.
246+
* Use the [`--prompt`](#testing-interactively) flag to inspect the output and
247+
verify each step.
248+
* Beware that some issues may get fixed and then regress multiple times. Try
249+
to keep the bisection range as close to the present day as possible. Compare
250+
the output of the "regressed" commit to the latest nightly to see if they
251+
are the same.
252+
* If the test only fails sporadically, use a [script](#testing-with-a-script)
253+
to run the compiler many times until it fails, or it passes enough
254+
iterations that you feel confident that it is good.
255+
* If the code requires relatively new language features, be careful not to
256+
pick a starting range that is too old.
257+
258+
[monotonically]: https://en.wikipedia.org/wiki/Bisection_(software_engineering)#Monotonicity
237259
[`cargo-bisect-rustc`]: https://github.com/rust-lang-nursery/cargo-bisect-rustc
238260
[issue #53157]: https://github.com/rust-lang/rust/issues/53157
239261
[issue #55036]: https://github.com/rust-lang/rust/issues/55036

0 commit comments

Comments
 (0)