@@ -73,9 +73,7 @@ important benefits:
73
73
out the coverage counts of each unique instantiation of a generic function,
74
74
if invoked with multiple type substitution variations.
75
75
76
- ## Components of LLVM Coverage Instrumentation in ` rustc `
77
-
78
- ### LLVM Runtime Dependency
76
+ ## The LLVM profiler runtime
79
77
80
78
Coverage data is only generated by running the executable Rust program. ` rustc `
81
79
statically links coverage-instrumented binaries with LLVM runtime code
@@ -94,209 +92,7 @@ When compiling with `-C instrument-coverage`,
94
92
[ compiler-rt-profile ] : https://github.com/llvm/llvm-project/tree/main/compiler-rt/lib/profile
95
93
[ crate-loader-postprocess ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/creader/struct.CrateLoader.html#method.postprocess
96
94
97
- ### MIR Pass: ` InstrumentCoverage `
98
-
99
- Coverage instrumentation is performed on the MIR with a [ MIR pass] [ mir-passes ]
100
- called [ ` InstrumentCoverage ` ] [ mir-instrument-coverage ] . This MIR pass analyzes
101
- the control flow graph (CFG)--represented by MIR ` BasicBlock ` s--to identify
102
- code branches, attaches [ ` FunctionCoverageInfo ` ] to the function's body,
103
- and injects additional [ ` Coverage ` ] [ coverage-statement ] statements into the
104
- ` BasicBlock ` s.
105
-
106
- A MIR ` Coverage ` statement is a virtual instruction that indicates a counter
107
- should be incremented when its adjacent statements are executed, to count
108
- a span of code ([ ` CodeRegion ` ] [ code-region ] ). It counts the number of times a
109
- branch is executed, and is referred to by coverage mappings in the function's
110
- coverage-info struct.
111
-
112
- Note that many coverage counters will _ not_ be converted into
113
- physical counters (or any other executable instructions) in the final binary.
114
- Some of them will be (see [ ` CoverageKind::CounterIncrement ` ] ),
115
- but other counters can be computed on the fly, when generating a coverage
116
- report, by mapping a ` CodeRegion ` to a coverage-counter _ expression_ .
117
-
118
- As an example:
119
-
120
- ``` rust
121
- fn some_func (flag : bool ) {
122
- // increment Counter(1)
123
- ...
124
- if flag {
125
- // increment Counter(2)
126
- ...
127
- } else {
128
- // count = Expression(1) = Counter(1) - Counter(2)
129
- ...
130
- }
131
- // count = Expression(2) = Counter(1) + Zero
132
- // or, alternatively, Expression(2) = Counter(2) + Expression(1)
133
- ...
134
- }
135
- ```
136
-
137
- In this example, four contiguous code regions are counted while only
138
- incrementing two counters.
139
-
140
- CFG analysis is used to not only determine _ where_ the branches are, for
141
- conditional expressions like ` if ` , ` else ` , ` match ` , and ` loop ` , but also to
142
- determine where expressions can be used in place of physical counters.
143
-
144
- The advantages of optimizing coverage through expressions are more pronounced
145
- with loops. Loops generally include at least one conditional branch that
146
- determines when to break out of a loop (a ` while ` condition, or an ` if ` or
147
- ` match ` with a ` break ` ). In MIR, this is typically lowered to a ` SwitchInt ` ,
148
- with one branch to stay in the loop, and another branch to break out of the
149
- loop. The branch that breaks out will almost always execute less often,
150
- so ` InstrumentCoverage ` chooses to add a ` CounterIncrement ` to that branch, and
151
- uses an expression (` Counter(loop) - Counter(break) ` ) for the branch that
152
- continues.
153
-
154
- The ` InstrumentCoverage ` MIR pass is documented in
155
- [ more detail below] [ instrument-coverage-pass-details ] .
156
-
157
- [ mir-passes ] : mir/passes.md
158
- [ mir-instrument-coverage ] : https://github.com/rust-lang/rust/tree/master/compiler/rustc_mir_transform/src/coverage
159
- [ `FunctionCoverageInfo` ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/mir/coverage/struct.FunctionCoverageInfo.html
160
- [ code-region ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_llvm/coverageinfo/ffi/struct.CodeRegion.html
161
- [ `CoverageKind::CounterIncrement` ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/mir/coverage/enum.CoverageKind.html#variant.CounterIncrement
162
- [ coverage-statement ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/mir/enum.StatementKind.html#variant.Coverage
163
- [ instrument-coverage-pass-details ] : #implementation-details-of-the-instrumentcoverage-mir-pass
164
-
165
- ### Counter Injection and Coverage Map Pre-staging
166
-
167
- When the compiler enters [ the Codegen phase] [ backend-lowering-mir ] , with a
168
- coverage-enabled MIR, [ ` codegen_statement() ` ] [ codegen-statement ] converts each
169
- MIR ` Statement ` into some backend-specific action or instruction.
170
- ` codegen_statement() ` forwards ` Coverage ` statements to
171
- [ ` codegen_coverage() ` ] [ codegen-coverage ] :
172
-
173
- ``` rust
174
- pub fn codegen_statement (& mut self , mut bx : Bx , statement : & mir :: Statement <'tcx >) -> Bx {
175
- ...
176
- match statement . kind {
177
- ...
178
- mir :: StatementKind :: Coverage (box ref coverage ) => {
179
- self . codegen_coverage (bx , coverage , statement . source_info. scope);
180
- }
181
- ```
182
-
183
- ` codegen_coverage() ` handles inlined statements and then forwards the coverage
184
- statement to [ ` Builder::add_coverage ` ] , which handles each ` CoverageKind ` as
185
- follows:
186
-
187
-
188
- - For both ` CounterIncrement ` and ` ExpressionUsed ` , the underlying counter or
189
- expression ID is passed through to the corresponding [ ` FunctionCoverage ` ]
190
- struct to indicate that the corresponding regions of code were not removed
191
- by MIR optimizations.
192
- - For ` CoverageKind::CounterIncrement ` s, an instruction is injected in the backend
193
- IR to increment the physical counter, by calling the ` BuilderMethod `
194
- [ ` instrprof_increment() ` ] [ instrprof-increment ] .
195
-
196
- ``` rust
197
- fn add_coverage (& mut self , instance : Instance <'tcx >, coverage : & Coverage ) {
198
- ...
199
- let Coverage { kind } = coverage ;
200
- match * kind {
201
- CoverageKind :: CounterIncrement { id } => {
202
- func_coverage . mark_counter_id_seen (id );
203
- ...
204
- bx . instrprof_increment (fn_name , hash , num_counters , index );
205
- }
206
- CoverageKind :: ExpressionUsed { id } => {
207
- func_coverage . mark_expression_id_seen (id );
208
- }
209
- }
210
- }
211
- ```
212
-
213
- > The function name ` instrprof_increment() ` is taken from the LLVM intrinsic
214
- call of the same name ([ ` llvm.instrprof.increment ` ] [ llvm-instrprof-increment ] ),
215
- and uses the same arguments and types; but note that, up to and through this
216
- stage (even though modeled after LLVM's implementation for code coverage
217
- instrumentation), the data and instructions are not strictly LLVM-specific.
218
- >
219
- > But since LLVM is the only Rust-supported backend with the tooling to
220
- process this form of coverage instrumentation, the backend for ` Coverage `
221
- statements is only implemented for LLVM, at this time.
222
-
223
- [ backend-lowering-mir ] : backend/lowering-mir.md
224
- [ codegen-statement ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_ssa/mir/struct.FunctionCx.html#method.codegen_statement
225
- [ codegen-coverage ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_ssa/mir/struct.FunctionCx.html#method.codegen_coverage
226
- [ `Builder::add_coverage` ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_llvm/builder/struct.Builder.html#method.add_coverage
227
- [ `FunctionCoverage` ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_llvm/coverageinfo/map_data/struct.FunctionCoverage.html
228
- [ instrprof-increment ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_ssa/traits/trait.BuilderMethods.html#tymethod.instrprof_increment
229
-
230
- ### Coverage Map Generation
231
-
232
- With the instructions to increment counters now implemented in LLVM IR,
233
- the last remaining step is to inject the LLVM IR variables that hold the
234
- static data for the coverage map.
235
-
236
- ` rustc_codegen_llvm ` 's [ ` compile_codegen_unit() ` ] [ compile-codegen-unit ] calls
237
- [ ` coverageinfo_finalize() ` ] [ coverageinfo-finalize ] ,
238
- which delegates its implementation to the
239
- [ ` rustc_codegen_llvm::coverageinfo::mapgen ` ] [ mapgen-finalize ] module.
240
-
241
- For each function ` Instance ` (code-generated from MIR, including multiple
242
- instances of the same MIR for generic functions that have different type
243
- substitution combinations), ` mapgen ` 's ` finalize() ` method queries the
244
- ` Instance ` -associated ` FunctionCoverage ` for its ` Counter ` s, ` Expression ` s,
245
- and ` CodeRegion ` s; and calls LLVM codegen APIs to generate
246
- properly-configured variables in LLVM IR, according to very specific
247
- details of the [ _ LLVM Coverage Mapping Format_ ] [ coverage-mapping-format ]
248
- (Version 6).[ ^ llvm-and-covmap-versions ]
249
-
250
- [ ^ llvm-and-covmap-versions ] : The Rust compiler (as of <!-- date-check: --> Nov 2024) supports _ LLVM Coverage Mapping Format_ 6.
251
- The Rust compiler will automatically use the most up-to-date coverage mapping format
252
- version that is compatible with the compiler's built-in version of LLVM.
253
-
254
- ``` rust
255
- pub fn finalize <'ll , 'tcx >(cx : & CodegenCx <'ll , 'tcx >) {
256
- ...
257
- if ! tcx . sess. instrument_coverage_except_unused_functions () {
258
- add_unused_functions (cx );
259
- }
260
-
261
- let mut function_coverage_map = match cx . coverage_context () {
262
- Some (ctx ) => ctx . take_function_coverage_map (),
263
- None => return ,
264
- };
265
- ...
266
- let mut mapgen = CoverageMapGenerator :: new ();
267
-
268
- for (instance , function_coverage ) in function_coverage_map {
269
- ...
270
- let coverage_mapping_buffer = llvm :: build_byte_buffer (| coverage_mapping_buffer | {
271
- mapgen . write_coverage_mapping (expressions , counter_regions , coverage_mapping_buffer );
272
- });
273
- ```
274
- _ code snippet trimmed for brevity_
275
-
276
- One notable first step performed by ` mapgen::finalize() ` is the call to
277
- [ ` add_unused_functions() ` ] [ add-unused-functions ] :
278
-
279
- When finalizing the coverage map, ` FunctionCoverage ` only has the ` CodeRegion ` s
280
- and counters for the functions that went through codegen; such as public
281
- functions and "used" functions (functions referenced by other "used" or public
282
- items). Any other functions (considered unused) were still parsed and processed
283
- through the MIR stage.
284
-
285
- The set of unused functions is computed via the set difference of all MIR
286
- ` DefId ` s (` tcx ` query ` mir_keys ` ) minus the codegenned ` DefId ` s (` tcx ` query
287
- ` codegened_and_inlined_items ` ). ` add_unused_functions() ` computes the set of
288
- unused functions, queries the ` tcx ` for the previously-computed ` CodeRegions ` ,
289
- for each unused MIR, synthesizes an LLVM function (with no internal statements,
290
- since it will not be called), and adds a new ` FunctionCoverage ` , with
291
- ` Unreachable ` code regions.
292
-
293
- [ compile-codegen-unit ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_llvm/base/fn.compile_codegen_unit.html
294
- [ coverageinfo-finalize ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_llvm/context/struct.CodegenCx.html#method.coverageinfo_finalize
295
- [ mapgen-finalize ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_llvm/coverageinfo/mapgen/fn.finalize.html
296
- [ coverage-mapping-format ] : https://llvm.org/docs/CoverageMappingFormat.html
297
- [ add-unused-functions ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_llvm/coverageinfo/mapgen/fn.add_unused_functions.html
298
-
299
- ## Testing LLVM Coverage
95
+ ## Testing coverage instrumentation
300
96
301
97
[ (See also the compiletest documentation for the ` tests/coverage `
302
98
test suite.)] ( ./tests/compiletest.md#coverage-tests )
@@ -341,151 +137,3 @@ and `mir-opt` tests can be refreshed by running:
341
137
[ `src/tools/coverage-dump` ] : https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump
342
138
[ `tests/coverage-run-rustdoc` ] : https://github.com/rust-lang/rust/tree/master/tests/coverage-run-rustdoc
343
139
[ `tests/codegen/instrument-coverage/testprog.rs` ] : https://github.com/rust-lang/rust/blob/master/tests/mir-opt/coverage/instrument_coverage.rs
344
-
345
- ## Implementation Details of the ` InstrumentCoverage ` MIR Pass
346
-
347
- The bulk of the implementation of the ` InstrumentCoverage ` MIR pass is performed
348
- by [ ` instrument_function_for_coverage ` ] . For each eligible MIR body, the instrumentor:
349
-
350
- - Prepares a [ coverage graph]
351
- - Extracts mapping information from MIR
352
- - Prepares counters for each relevant node/edge in the coverage graph
353
- - Creates mapping data to be embedded in side-tables attached to the MIR body
354
- - Injects counters and other coverage statements into MIR
355
-
356
- The [ coverage graph] is a coverage-specific simplification of the MIR control
357
- flow graph (CFG). Its nodes are [ ` BasicCoverageBlock ` s] [ bcb ] , which
358
- encompass one or more sequentially-executed MIR ` BasicBlock ` s
359
- (with no internal branching).
360
-
361
- Nodes and edges in the graph can have associated [ ` BcbCounter ` ] s, which are
362
- stored in [ ` CoverageCounters ` ] .
363
-
364
- [ `instrument_function_for_coverage` ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/coverage/fn.instrument_function_for_coverage.html
365
- [ coverage graph ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/coverage/graph/struct.CoverageGraph.html
366
- [ bcb ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/coverage/graph/struct.BasicCoverageBlock.html
367
- [ `BcbCounter` ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/coverage/counters/enum.BcbCounter.html
368
- [ `CoverageCounters` ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/coverage/counters/struct.CoverageCounters.html
369
-
370
- ### The ` CoverageGraph `
371
-
372
- The [ ` CoverageGraph ` ] [ coverage graph ] is derived from the MIR (` mir::Body ` ).
373
-
374
- ``` rust
375
- let basic_coverage_blocks = CoverageGraph :: from_mir (mir_body );
376
- ```
377
-
378
- Like ` mir::Body ` , the ` CoverageGraph ` is also a
379
- [ ` DirectedGraph ` ] [ directed-graph ] . Both graphs represent the function's
380
- fundamental control flow, with many of the same
381
- [ ` graph trait ` ] [ graph-traits ] s, supporting ` start_node() ` , ` num_nodes() ` ,
382
- ` successors() ` , ` predecessors() ` , and ` is_dominated_by() ` .
383
-
384
- For anyone that knows how to work with the [ MIR, as a CFG] [ mir-dev-guide ] , the
385
- ` CoverageGraph ` will be familiar, and can be used in much the same way.
386
- The nodes of the ` CoverageGraph ` are ` BasicCoverageBlock ` s (BCBs), which
387
- index into an ` IndexVec ` of ` BasicCoverageBlockData ` . This is analogous
388
- to the MIR CFG of ` BasicBlock ` s that index ` BasicBlockData ` .
389
-
390
- Each ` BasicCoverageBlockData ` captures one or more MIR ` BasicBlock ` s,
391
- exclusively, and represents the maximal-length sequence of ` BasicBlocks `
392
- without conditional branches.
393
-
394
- [ ` compute_basic_coverage_blocks() ` ] [ compute-basic-coverage-blocks ] builds the
395
- ` CoverageGraph ` as a coverage-specific simplification of the MIR CFG. In
396
- contrast with the [ ` SimplifyCfg ` ] [ simplify-cfg ] MIR pass, this step does
397
- not alter the MIR itself, because the ` CoverageGraph ` aggressively simplifies
398
- the CFG, and ignores nodes that are not relevant to coverage. For example:
399
-
400
- - The BCB CFG ignores (excludes) branches considered not relevant
401
- to the current coverage solution. It excludes unwind-related code[ ^ 78544 ]
402
- that is injected by the Rust compiler but has no physical source
403
- code to count, which allows a ` Call ` -terminated BasicBlock
404
- to be merged with its successor, within a single BCB.
405
- - A ` Goto ` -terminated ` BasicBlock ` can be merged with its successor
406
- ** _ as long as_ ** it has the only incoming edge to the successor
407
- ` BasicBlock ` .
408
- - Some BasicBlock terminators support Rust-specific concerns--like
409
- borrow-checking--that are not relevant to coverage analysis. ` FalseUnwind ` ,
410
- for example, can be treated the same as a ` Goto ` (potentially merged with
411
- its successor into the same BCB).
412
-
413
- [ ^ 78544 ] : (Note, however, that Issue [ #78544 ] [ rust-lang/rust#78544 ] considers
414
- providing future support for coverage of programs that intentionally
415
- ` panic ` , as an option, with some non-trivial cost.)
416
-
417
- The BCB CFG is critical to simplifying the coverage analysis by ensuring graph path-based
418
- queries (` is_dominated_by() ` , ` predecessors ` , ` successors ` , etc.) have branch (control flow)
419
- significance.
420
-
421
- [ directed-graph ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/graph/trait.DirectedGraph.html
422
- [ graph-traits ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/graph/index.html#traits
423
- [ mir-dev-guide ] : mir/index.md
424
- [ compute-basic-coverage-blocks ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/coverage/graph/struct.CoverageGraph.html#method.compute_basic_coverage_blocks
425
- [ simplify-cfg ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/simplify/enum.SimplifyCfg.html
426
- [ rust-lang/rust#78544 ] : https://github.com/rust-lang/rust/issues/78544
427
-
428
- ### ` make_bcb_counters() `
429
-
430
- [ ` make_bcb_counters ` ] traverses the ` CoverageGraph ` and adds a
431
- ` Counter ` or ` Expression ` to every BCB. It uses _ Control Flow Analysis_
432
- to determine where an ` Expression ` can be used in place of a ` Counter ` .
433
- ` Expressions ` have no runtime overhead, so if a viable expression (adding or
434
- subtracting two other counters or expressions) can compute the same result as
435
- an embedded counter, an ` Expression ` is preferred.
436
-
437
- [ ` TraverseCoverageGraphWithLoops ` ] [ traverse-coverage-graph-with-loops ]
438
- provides a traversal order that ensures all ` BasicCoverageBlock ` nodes in a
439
- loop are visited before visiting any node outside that loop. The traversal
440
- state includes a ` context_stack ` , with the current loop's context information
441
- (if in a loop), as well as context for nested loops.
442
-
443
- Within loops, nodes with multiple outgoing edges (generally speaking, these
444
- are BCBs terminated in a ` SwitchInt ` ) can be optimized when at least one
445
- branch exits the loop and at least one branch stays within the loop. (For an
446
- ` if ` or ` while ` , there are only two branches, but a ` match ` may have more.)
447
-
448
- A branch that does not exit the loop should be counted by ` Expression ` , if
449
- possible. Note that some situations require assigning counters to BCBs before
450
- they are visited by traversal, so the ` counter_kind ` (` CoverageKind ` for
451
- a ` Counter ` or ` Expression ` ) may have already been assigned, in which case
452
- one of the other branches should get the ` Expression ` .
453
-
454
- For a node with more than two branches (such as for more than two
455
- ` match ` patterns), only one branch can be optimized by ` Expression ` . All
456
- others require a ` Counter ` (unless its BCB ` counter_kind ` was previously
457
- assigned).
458
-
459
- A branch expression is derived from the equation:
460
-
461
- ``` text
462
- Counter(branching_node) = SUM(Counter(branches))
463
- ```
464
-
465
- It's important to
466
- be aware that the ` branches ` in this equation are the outgoing _ edges_
467
- from the ` branching_node ` , but a ` branch ` 's target node may have other
468
- incoming edges. Given the following graph, for example, the count for
469
- ` B ` is the sum of its two incoming edges:
470
-
471
- <img alt="Example graph with multiple incoming edges to a branch node"
472
- src="img/coverage-branch-counting-01.png" class="center" style="width: 25%">
473
- <br />
474
-
475
- In this situation, BCB node ` B ` may require an edge counter for its
476
- "edge from A", and that edge might be computed from an ` Expression ` ,
477
- ` Counter(A) - Counter(C) ` . But an expression for the BCB _ node_ ` B `
478
- would be the sum of all incoming edges:
479
-
480
- ``` text
481
- Expression((Counter(A) - Counter(C)) + SUM(Counter(remaining_edges)))
482
- ```
483
-
484
- Note that this is only one possible configuration. The actual choice
485
- of ` Counter ` vs. ` Expression ` also depends on the order of counter
486
- assignments, and whether a BCB or incoming edge counter already has
487
- its ` Counter ` or ` Expression ` .
488
-
489
- [ `make_bcb_counters` ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/coverage/counters/struct.CoverageCounters.html#method.make_bcb_counters
490
- [ bcb-counters ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/coverage/counters/struct.BcbCounters.html
491
- [ traverse-coverage-graph-with-loops ] : https://doc.rust-lang.org/nightly/nightly-rustc/rustc_mir_transform/coverage/graph/struct.TraverseCoverageGraphWithLoops.html
0 commit comments