@@ -233,7 +233,29 @@ cargo-bisect-rustc --script=./test.sh --test-dir=foo \
233
233
--end=2c2e2c57dc2140cfb62a8abb9312b89f02c59f3c
234
234
```
235
235
236
-
236
+ ## Varying tests
237
+
238
+ When writing your test and picking a bisection range, you should be careful to
239
+ ensure that the test won't vary between pass/fail over time. It should only
240
+ transition from good to bad once in the bisection range (it must change
241
+ [ monotonically] ). The following are some suggestions for dealing with a
242
+ potentially varying test:
243
+
244
+ * Use the ` -vv ` flag (very verbose) to display the output from the compiler to
245
+ make sure it is what you expect.
246
+ * Use the [ ` --prompt ` ] ( #testing-interactively ) flag to inspect the output and
247
+ verify each step.
248
+ * Beware that some issues may get fixed and then regress multiple times. Try
249
+ to keep the bisection range as close to the present day as possible. Compare
250
+ the output of the "regressed" commit to the latest nightly to see if they
251
+ are the same.
252
+ * If the test only fails sporadically, use a [ script] ( #testing-with-a-script )
253
+ to run the compiler many times until it fails, or it passes enough
254
+ iterations that you feel confident that it is good.
255
+ * If the code requires relatively new language features, be careful not to
256
+ pick a starting range that is too old.
257
+
258
+ [ monotonically ] : https://en.wikipedia.org/wiki/Bisection_(software_engineering)#Monotonicity
237
259
[ `cargo-bisect-rustc` ] : https://github.com/rust-lang-nursery/cargo-bisect-rustc
238
260
[ issue #53157 ] : https://github.com/rust-lang/rust/issues/53157
239
261
[ issue #55036 ] : https://github.com/rust-lang/rust/issues/55036
0 commit comments