From ce71058a20aa8643685eb23df54f5ecad04be353 Mon Sep 17 00:00:00 2001 From: Zalathar Date: Mon, 11 Sep 2023 11:52:21 +1000 Subject: [PATCH 1/2] Document the `coverage-map` and `run-coverage` test suites --- src/tests/compiletest.md | 55 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/src/tests/compiletest.md b/src/tests/compiletest.md index 9c969bff9..2b02cae69 100644 --- a/src/tests/compiletest.md +++ b/src/tests/compiletest.md @@ -67,6 +67,12 @@ The following test suites are available, with links for more information: - `run-make-fulldeps` — `run-make` tests which require a linkable build of `rustc`, or the rust demangler - [`run-pass-valgrind`](#valgrind-tests) — tests run with Valgrind +- [`coverage-map`](#coverage-tests) - tests for coverage maps produced by + coverage instrumentation +- [`run-coverage`](#coverage-tests) - tests that run an instrumented program + and check its coverage report +- [`run-coverage-rustdoc`](#coverage-tests) - coverage tests that also run + instrumented doctests - [Rustdoc tests](../rustdoc.md#tests): - `rustdoc` — tests for rustdoc, making sure that the generated files contain the expected documentation. @@ -394,6 +400,55 @@ These may be removed in the future. [`tests/run-pass-valgrind`]: https://github.com/rust-lang/rust/tree/master/tests/run-pass-valgrind +### Coverage tests + +The tests in [`tests/coverage-map`] test the mappings between source code +regions and coverage counters that are emitted by LLVM. +They compile the test with `--emit=llvm-ir`, +then use a custom tool ([`src/tools/coverage-dump`]) +to extract and pretty-print the coverage mappings embedded in the IR. +These tests don't require the profiler runtime, so they run in PR CI jobs and +are easy to run/bless locally. + +These coverage map tests can be sensitive to changes in MIR lowering or MIR +optimizations, producing mappings that are different but produce identical +coverage reports. + +As a rule of thumb, any PR that doesn't change coverage-specific +code should **feel free to re-bless** the `coverage-map` tests as necessary, +without worrying about the actual changes, as long as the `run-coverage` tests +still pass. + +--- + +The tests in [`tests/run-coverage`] perform an end-to-end test of coverage reporting. +They compile a test program with coverage instrumentation, run that program to +produce raw coverage data, and then use LLVM tools to process that data into a +human-readable code coverage report. + +Instrumented binaries need to be linked against the LLVM profiler runtime, +so `run-coverage` tests are **automatically skipped** +unless the profiler runtime is enabled in `config.toml`: + +```toml +# config.toml +[build] +profiler = true +``` + +This also means that they typically don't run in PR CI jobs, +though they do run in the full set of CI jobs used for merging. + +The tests in [`tests/run-coverage-rustdoc`] also run instrumented doctests and +include them in the coverage report. This avoids having to build rustdoc when +only running the main `run-coverage` suite. + +[`tests/coverage-map`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-map +[`src/tools/coverage-dump`]: https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump +[`tests/run-coverage`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage +[`tests/run-coverage-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage-rustdoc + + ## Building auxiliary crates It is common that some tests require additional auxiliary crates to be compiled. From 2c937ce8dcbc41dbca6696e9c0a2fa2ffc4bb08b Mon Sep 17 00:00:00 2001 From: Zalathar Date: Mon, 11 Sep 2023 12:28:54 +1000 Subject: [PATCH 2/2] Revise the docs for coverage instrumentation tests --- src/llvm-coverage-instrumentation.md | 32 ++++++++++++++++++---------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/src/llvm-coverage-instrumentation.md b/src/llvm-coverage-instrumentation.md index dc18a0121..7147e5dbb 100644 --- a/src/llvm-coverage-instrumentation.md +++ b/src/llvm-coverage-instrumentation.md @@ -274,13 +274,22 @@ since it will not be called), and adds a new `FunctionCoverage`, with ## Testing LLVM Coverage +[(See also the compiletest documentation for the `tests/coverage-map` and +`tests/run-coverage` test suites.)](./tests/compiletest.md#coverage-tests) + Coverage instrumentation in the MIR is validated by a `mir-opt` test: -[`instrument-coverage`][mir-opt-test]. +[`tests/mir-opt/instrument_coverage.rs`]. + +Coverage instrumentation in LLVM IR is validated by the [`tests/coverage-map`] +test suite. These tests compile a test program to LLVM IR assembly, and then +use the [`src/tools/coverage-dump`] tool to extract and pretty-print the +coverage mappings that would be embedded in the final binary. -More complete testing of end-to-end coverage instrumentation and reports are -done in the `run-make-fulldeps` tests, with sample Rust programs (to be -instrumented) in the [`tests/run-coverage`] directory, -together with the actual tests and expected results. +End-to-end testing of coverage instrumentation and coverage reporting is +performed by the [`tests/run-coverage`] and [`tests/run-coverage-rustdoc`] +test suites. These tests compile and run a test program with coverage +instrumentation, then use LLVM tools to convert the coverage data into a +human-readable coverage report. Finally, the [`coverage-llvmir`] test compiles a simple Rust program with `-C instrument-coverage` and compares the compiled program's LLVM IR to @@ -288,18 +297,19 @@ expected LLVM IR instructions and structured data for a coverage-enabled program, including various checks for Coverage Map-related metadata and the LLVM intrinsic calls to increment the runtime counters. -Expected results for both the `mir-opt` tests and the `coverage*` tests -can be refreshed by running: +Expected results for the `coverage-map`, `run-coverage`, `run-coverage-rustdoc`, +and `mir-opt` tests can be refreshed by running: ```shell +./x test tests/*coverage* --bless ./x test tests/mir-opt --bless -./x test tests/run-coverage --bless -./x test tests/run-coverage-rustdoc --bless ``` -[mir-opt-test]: https://github.com/rust-lang/rust/blob/master/tests/mir-opt/instrument_coverage.rs +[`tests/mir-opt/instrument_coverage.rs`]: https://github.com/rust-lang/rust/blob/master/tests/mir-opt/instrument_coverage.rs +[`tests/coverage-map`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-map +[`src/tools/coverage-dump`]: https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump [`tests/run-coverage`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage -[spanview-debugging]: compiler-debugging.md#viewing-spanview-output +[`tests/run-coverage-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/run-coverage-rustdoc [`coverage-llvmir`]: https://github.com/rust-lang/rust/tree/master/tests/run-make/coverage-llvmir ## Implementation Details of the `InstrumentCoverage` MIR Pass