|
1 | | -# Bisecting Regressions |
| 1 | +# Tutorial |
2 | 2 |
|
3 | | -The [`cargo-bisect-rustc`] tool makes it super easy to find exactly when |
4 | | -behavior has regressed in rustc. It automatically downloads rustc |
5 | | -artifacts and tests them against a project you provide until it finds |
6 | | -the regression. |
7 | | - |
8 | | -## Installation |
9 | | - |
10 | | -If you're going to bisect for linux-musl host, install musl toolchain and run: |
11 | | - |
12 | | -```sh |
13 | | -cargo install cargo-bisect-rustc --features git2/vendored-openssl |
14 | | -``` |
15 | | - |
16 | | -Otherwise, run: |
17 | | - |
18 | | -```sh |
19 | | -cargo install cargo-bisect-rustc |
20 | | -``` |
21 | | - |
22 | | -or (to avoid cloning rustc) |
23 | | - |
24 | | -``` |
25 | | -RUST_SRC_REPO=/path/to/rust cargo install cargo-bisect-rustc |
26 | | -``` |
27 | | - |
28 | | -The `RUST_SRC_REPO` should be a path to a git clone of the rust repo. |
29 | | -The current order how `cargo-bisect-rustc` finds Rust repository path is: |
30 | | -* `RUST_SRC_REPO` at runtime. |
31 | | -* `rust.git` in current dir at runtime. |
32 | | -* `RUST_SRC_REPO` that was set at compilation time. |
33 | | -* Clone https://github.com/rust-lang/rust automatically |
34 | | - (only necessary if doing git hash bisections). |
35 | | - |
36 | | -First, if you have a nightly version of the compiler already installed |
37 | | -as the default toolchain and you don't pass an end flag, the tool is |
38 | | -going to assume that that's the version that has regressed and use it as |
39 | | -the "bad" version. Otherwise would use that start point or just use the |
40 | | -latest nightly. |
41 | | -If you have provided a start flag it would use that as the "good" |
42 | | -version, otherwise is going to search for a good one backwards. |
43 | | - |
44 | | -Once the tool has an start point (good version) and end point (bad |
45 | | -version), is going to do the bisect to find the regressed nightly. Once |
46 | | -it finds the regressed nightly is going to look between that nightly and |
47 | | -the one of the day before for the specific commit that has the |
48 | | -regression and report everything back to the user. |
49 | | - |
50 | | -So, there's a bunch of ways to run the tool, the easiest one is to |
51 | | -allow the tool to do the job and just run `cargo bisect-rustc` on the |
52 | | -project and let the tool figure things out. |
53 | | - |
54 | | -## Finding a regression |
55 | | - |
56 | | -Create a cargo project that demonstrates the regression. Let's use |
57 | | -[issue #53157] as an example: |
58 | | - |
59 | | -``` |
60 | | -cargo new foo |
61 | | -``` |
62 | | - |
63 | | -Edit `foo/src/main.rs` with the example from the issue: |
64 | | - |
65 | | -```rust |
66 | | -macro_rules! m { |
67 | | - () => {{ |
68 | | - fn f(_: impl Sized) {} |
69 | | - f |
70 | | - }} |
71 | | -} |
72 | | - |
73 | | -fn main() { |
74 | | - fn f() -> impl Sized {}; |
75 | | - m!()(f()); |
76 | | -} |
77 | | -``` |
78 | | - |
79 | | -Then run `cargo bisect-rustc --end=2018-08-04`. |
80 | | - |
81 | | -We need to provide the end point for this particular example because |
82 | | -that's an old regression already fixed on the latests nightlies. |
83 | | -We could also provide a start point if we know one, that's going to make |
84 | | -the tool avoid searching for that so answer our request faster. |
85 | | -For instance: |
86 | | - |
87 | | -``` |
88 | | -cargo bisect-rustc --test-dir=foo --start=2018-05-07 --end=2018-08-04 |
89 | | -``` |
90 | | - |
91 | | -By default it will run `cargo build` in the project and check whether or not |
92 | | -it fails. You can also use the flag `--regress` to specify other common |
93 | | -regression criteria, e.g. `--regress=ice` for internal compiler errors. |
94 | | - |
95 | | -In out example, in just a few steps, we can we find that it stopped working on |
96 | | -`nightly-2018-07-30`. |
97 | | - |
98 | | -> *Note:* Consider using the `--preserve` flag to keep the downloaded |
99 | | -> artifacts for future runs. They are stored in the normal location for your |
100 | | -> toolchains in `RUSTUP_HOME`. |
101 | | -
|
102 | | -After that is going to automatically search for the commit that |
103 | | -introduced the regression. |
104 | | - |
105 | | -## Finding a regression between commits |
106 | | - |
107 | | -We can also just ask the tool to look between commits if that's what we |
108 | | -want. As long as the regression wasn't too long ago, we can find the |
109 | | -exact PR that caused the regression. Use git hashes from the rustc |
110 | | -repo's log as the start/end parameters. They must be from bors on the |
111 | | -master branch. |
112 | | - |
113 | | -To find a list of all such usable commit hashes, we can use `git log` in the |
114 | | -`RUST_SRC_REPO` git clone. After regressing to a nightly, and padding a couple |
115 | | -days before and after its date to allow for the CI build process time: |
116 | | - |
117 | | -``` |
118 | | -git log --since "JUL 28 2018" --until "JUL 30 2018" --author=bors --pretty=format:"%H %an %ad" |
119 | | -``` |
120 | | - |
121 | | -will show |
122 | | - |
123 | | -``` |
124 | | -e4378412ecfc2a4ff5dfd65fef53fa6be691f689 bors Mon Jul 30 10:19:38 2018 +0000 |
125 | | -5ed2b5120bd875a7eb9fd8545d86eb1de1e41bce bors Mon Jul 30 08:25:36 2018 +0000 |
126 | | -7bbcd005b30582d07f1a39dcf50f77b54e055828 bors Mon Jul 30 06:29:39 2018 +0000 |
127 | | -a3f519df09bf40d09c1a111599b8f115f11fbb49 bors Mon Jul 30 04:34:19 2018 +0000 |
128 | | -b12235db096ab24a31e6e894757abfe8b018d44a bors Mon Jul 30 01:08:13 2018 +0000 |
129 | | -866a713258915e6cbb212d135f751a6a8c9e1c0a bors Sun Jul 29 21:37:47 2018 +0000 |
130 | | -70cac59031d5c33962a1f53cdca9359c0dcd1f9f bors Sun Jul 29 19:37:28 2018 +0000 |
131 | | -75af9df71b9eea84f281cf7de72c3e3cc2b02222 bors Sun Jul 29 13:23:01 2018 +0000 |
132 | | -2a9dc245c60ab4478b3bc4670aaad4b39e646366 bors Sun Jul 29 11:27:48 2018 +0000 |
133 | | -023fd7e74a9eb5bafcb75fcbe69b7110e9de4492 bors Sun Jul 29 09:33:37 2018 +0000 |
134 | | -a5c2d0fffaaf0b764c01bc4066e51ffd475ceae9 bors Sun Jul 29 06:32:24 2018 +0000 |
135 | | -fb0653e40289eecf32f3fac1e84fc69b815ce5cb bors Sun Jul 29 03:20:54 2018 +0000 |
136 | | -6a2c97c38d297307dd8554853890f51144f62172 bors Sun Jul 29 01:14:39 2018 +0000 |
137 | | -6323d9a45bdf0ac2a9319a6a558537e0a7e6abd1 bors Sat Jul 28 23:10:10 2018 +0000 |
138 | | -dab71516f1f4f6a63e32dffeb2625a12e5113485 bors Sat Jul 28 20:44:17 2018 +0000 |
139 | | -4234adf0d4fa56e8a8b8d790fb4992d160ab2188 bors Sat Jul 28 18:41:40 2018 +0000 |
140 | | -d75458200516f06455d175adc001fd993d674050 bors Sat Jul 28 16:44:21 2018 +0000 |
141 | | -26e73dabeb7a15e0e38feb2cadca3c1f740a61d2 bors Sat Jul 28 14:26:16 2018 +0000 |
142 | | -5b465e309da475aaedcb742ef29094c82e970051 bors Sat Jul 28 11:37:41 2018 +0000 |
143 | | -``` |
144 | | - |
145 | | -and we can, for example, pick the last commit on the day before the nightly, |
146 | | -`6323d9a45bdf0ac2a9319a6a558537e0a7e6abd1`, as the start of the range, and the |
147 | | -last commit on the day of the nightly, `866a713258915e6cbb212d135f751a6a8c9e1c0a`, |
148 | | -as the end of the range. |
149 | | - |
150 | | -Assuming you aren't reading this too far in the future, the |
151 | | -following should work: |
152 | | - |
153 | | -``` |
154 | | -cargo bisect-rustc --test-dir=foo \ |
155 | | - --start=6323d9a45bdf0ac2a9319a6a558537e0a7e6abd1 \ |
156 | | - --end=866a713258915e6cbb212d135f751a6a8c9e1c0a |
157 | | -``` |
158 | | - |
159 | | -This tells us that the regression started with |
160 | | -`70cac59031d5c33962a1f53cdca9359c0dcd1f9f` and you can look at the git log to |
161 | | -find the PR. Here, #51361. |
162 | | - |
163 | | -``` |
164 | | - git log -1 70cac59031d5c33962a1f53cdca9359c0dcd1f9f |
165 | | -``` |
166 | | - |
167 | | -shows the merge commit's description starts with "`Auto merge of #51361`". |
168 | | - |
169 | | -## Testing interactively |
170 | | - |
171 | | -Pass/fail of `cargo build` may not be what you're after. Perhaps the issue is |
172 | | -an error message changed, so both the "good" and "bad" version will fail to |
173 | | -compile, just with a different message. Or maybe something used to fail, and |
174 | | -now erroneously passes. You can use the interactive feature with the |
175 | | -`--prompt` flag to visually inspect a build and tell `cargo-bisect-rustc` |
176 | | -what's "good" and what's "bad". Let's use [issue #55036] as an example where |
177 | | -an error message changed: |
178 | | - |
179 | | -`foo/src/main.rs`: |
180 | | -```rust |
181 | | -struct Foo { |
182 | | - bar: i32 |
183 | | -} |
184 | | - |
185 | | -trait Baz { |
186 | | - fn f(Foo { bar }: Foo) {} |
187 | | -} |
188 | | -``` |
189 | | - |
190 | | -This historically emitted a bad error, was updated to emit a nice error (E0642 |
191 | | -added in #53051), but then that nice error was lost somewhere (on the 2015 |
192 | | -edition). Let's find where it was lost! Grab the ranges between where it was |
193 | | -added and where we know it fails: |
194 | | - |
195 | | -``` |
196 | | -cargo bisect-rustc --prompt --test-dir=foo \ |
197 | | - --start=ab93561b5fa54954159480ddc10bbb69f015e539 \ |
198 | | - --end=2c2e2c57dc2140cfb62a8abb9312b89f02c59f3c |
199 | | -``` |
200 | | - |
201 | | -At each step, `cargo-bisect-rustc` will show the output and ask you: |
202 | | - |
203 | | -``` |
204 | | -ab93561b5fa54954159480ddc10bbb69f015e539 finished with exit code Some(101). |
205 | | -please select an action to take: |
206 | | -> mark regressed |
207 | | - mark baseline |
208 | | - retry |
209 | | -``` |
210 | | - |
211 | | -Choose `mark baseline` with the nice E0642 message, and `mark regressed` with |
212 | | -the less-favorable token error. Fairly quickly we find it regressed in |
213 | | -af50e3822c4ceda60445c4a2adbb3bfa480ebd39 which is a rollup merge. However, |
214 | | -it's not too hard to look through the commits and find a likely culprit. |
215 | | - |
216 | | -## Testing with a script |
217 | | - |
218 | | -Using the `--script` option allows you to do something more fancy than just |
219 | | -`cargo build`. Maybe you need to run cargo multiple times, or just call |
220 | | -`rustc` directly, or you want to automatically grep through the output. The |
221 | | -possibilities are endless! Just write a little shell script that exits 0 for |
222 | | -the baseline, and exits nonzero for the regression. As an example, the |
223 | | -previous interactive session can be hands-free automated with this script: |
224 | | - |
225 | | -`foo/test.sh`: |
226 | | -```sh |
227 | | -#!/bin/sh |
228 | | - |
229 | | -# Fail if we no longer get a `E0642` error: |
230 | | -cargo check 2>&1 | grep E0642 |
231 | | -``` |
232 | | - |
233 | | -And then run: |
234 | | - |
235 | | -``` |
236 | | -cargo bisect-rustc --script=./test.sh --test-dir=foo \ |
237 | | - --start=ab93561b5fa54954159480ddc10bbb69f015e539 \ |
238 | | - --end=2c2e2c57dc2140cfb62a8abb9312b89f02c59f3c |
239 | | -``` |
240 | | - |
241 | | -## Varying tests |
242 | | - |
243 | | -When writing your test and picking a bisection range, you should be careful to |
244 | | -ensure that the test won't vary between pass/fail over time. It should only |
245 | | -transition from good to bad once in the bisection range (it must change |
246 | | -[monotonically]). The following are some suggestions for dealing with a |
247 | | -potentially varying test: |
248 | | - |
249 | | -* Use the `-vv` flag (very verbose) to display the output from the compiler to |
250 | | - make sure it is what you expect. |
251 | | -* Use the [`--prompt`](#testing-interactively) flag to inspect the output and |
252 | | - verify each step. |
253 | | -* Beware that some issues may get fixed and then regress multiple times. Try |
254 | | - to keep the bisection range as close to the present day as possible. Compare |
255 | | - the output of the "regressed" commit to the latest nightly to see if they |
256 | | - are the same. |
257 | | -* If the test only fails sporadically, use a [script](#testing-with-a-script) |
258 | | - to run the compiler many times until it fails, or it passes enough |
259 | | - iterations that you feel confident that it is good. |
260 | | -* If the code requires relatively new language features, be careful not to |
261 | | - pick a starting range that is too old. |
262 | | - |
263 | | -[monotonically]: https://en.wikipedia.org/wiki/Bisection_(software_engineering)#Monotonicity |
264 | | -[`cargo-bisect-rustc`]: https://github.com/rust-lang/cargo-bisect-rustc |
265 | | -[issue #53157]: https://github.com/rust-lang/rust/issues/53157 |
266 | | -[issue #55036]: https://github.com/rust-lang/rust/issues/55036 |
| 3 | +The tutorial has moved to the new guide at <https://rust-lang.github.io/cargo-bisect-rustc/>. |
0 commit comments