Rollup merge of rust-lang#125005 - RalfJung:miri-sync, r=RalfJung

Miri subtree update

r? `@ghost`

Author: matthiaskrgr

src/tools/miri/.gitignore

@@ -9,5 +9,5 @@ tex/*/out
 perf.data
 perf.data.old
 flamegraph.svg
-tests/extern-so/libtestlib.so
+tests/native-lib/libtestlib.so
 .auto-*

src/tools/miri/CONTRIBUTING.md

@@ -72,14 +72,14 @@ For example:
 
 You can (cross-)run the entire test suite using:
 
-```
+```sh
 ./miri test
-MIRI_TEST_TARGET=i686-unknown-linux-gnu ./miri test
+./miri test --target i686-unknown-linux-gnu
 ```
 
 `./miri test FILTER` only runs those tests that contain `FILTER` in their filename (including the
-base directory, e.g. `./miri test fail` will run all compile-fail tests). These filters are passed
-to `cargo test`, so for multiple filters you need to use `./miri test -- FILTER1 FILTER2`.
+base directory, e.g. `./miri test fail` will run all compile-fail tests). Multiple filters are
+supported: `./miri test FILTER1 FILTER2` runs all tests that contain either string.
 
 #### Fine grained logging
 
@@ -139,9 +139,8 @@ and then you can use it as if it was installed by `rustup` as a component of the
 in the `miri` toolchain's sysroot to prevent conflicts with other toolchains.
 The Miri binaries in the `cargo` bin directory (usually `~/.cargo/bin`) are managed by rustup.
 
-There's a test for the cargo wrapper in the `test-cargo-miri` directory; run
-`./run-test.py` in there to execute it. Like `./miri test`, this respects the
-`MIRI_TEST_TARGET` environment variable to execute the test for another target.
+There's a test for the cargo wrapper in the `test-cargo-miri` directory; run `./run-test.py` in
+there to execute it. You can pass `--target` to execute the test for another target.
 
 ### Using a modified standard library
 
@@ -287,3 +286,41 @@ https. Add the following to your `.gitconfig`:
 [url "git@github.com:"]
     pushInsteadOf = https://github.com/
 ```
+
+## Internal environment variables
+
+The following environment variables are *internal* and must not be used by
+anyone but Miri itself. They are used to communicate between different Miri
+binaries, and as such worth documenting:
+
+* `CARGO_EXTRA_FLAGS` is understood by `./miri` and passed to all host cargo invocations.
+* `MIRI_BE_RUSTC` can be set to `host` or `target`. It tells the Miri driver to
+  actually not interpret the code but compile it like rustc would. With `target`, Miri sets
+  some compiler flags to prepare the code for interpretation; with `host`, this is not done.
+  This environment variable is useful to be sure that the compiled `rlib`s are compatible
+  with Miri.
+* `MIRI_CALLED_FROM_SETUP` is set during the Miri sysroot build,
+  which will re-invoke `cargo-miri` as the `rustc` to use for this build.
+* `MIRI_CALLED_FROM_RUSTDOC` when set to any value tells `cargo-miri` that it is
+  running as a child process of `rustdoc`, which invokes it twice for each doc-test
+  and requires special treatment, most notably a check-only build before interpretation.
+  This is set by `cargo-miri` itself when running as a `rustdoc`-wrapper.
+* `MIRI_CWD` when set to any value tells the Miri driver to change to the given
+  directory after loading all the source files, but before commencing
+  interpretation. This is useful if the interpreted program wants a different
+  working directory at run-time than at build-time.
+* `MIRI_LOCAL_CRATES` is set by `cargo-miri` to tell the Miri driver which
+  crates should be given special treatment in diagnostics, in addition to the
+  crate currently being compiled.
+* `MIRI_ORIG_RUSTDOC` is set and read by different phases of `cargo-miri` to remember the
+  value of `RUSTDOC` from before it was overwritten.
+* `MIRI_REPLACE_LIBRS_IF_NOT_TEST` when set to any value enables a hack that helps bootstrap
+  run the standard library tests in Miri.
+* `MIRI_TEST_TARGET` is set by `./miri test` (and `./x.py test miri`) to tell the test harness about
+  the chosen target.
+* `MIRI_VERBOSE` when set to any value tells the various `cargo-miri` phases to
+  perform verbose logging.
+* `MIRI_HOST_SYSROOT` is set by bootstrap to tell `cargo-miri` which sysroot to use for *host*
+  operations.
+* `RUSTC_BLESS` is set by `./miri test` (and `./x.py test miri`) to indicate bless-mode to the test
+  harness.

src/tools/miri/README.md

@@ -1,39 +1,34 @@
 # Miri
 
-An experimental interpreter for [Rust][rust]'s
-[mid-level intermediate representation][mir] (MIR). It can run binaries and
-test suites of cargo projects and detect certain classes of
-[undefined behavior](https://doc.rust-lang.org/reference/behavior-considered-undefined.html),
-for example:
+Miri is an [Undefined Behavior][reference-ub] detection tool for Rust. It can run binaries and test
+suites of cargo projects and detect unsafe code that fails to uphold its safety requirements. For
+instance:
 
 * Out-of-bounds memory accesses and use-after-free
 * Invalid use of uninitialized data
 * Violation of intrinsic preconditions (an [`unreachable_unchecked`] being
   reached, calling [`copy_nonoverlapping`] with overlapping ranges, ...)
 * Not sufficiently aligned memory accesses and references
-* Violation of *some* basic type invariants (a `bool` that is not 0 or 1, for example,
+* Violation of basic type invariants (a `bool` that is not 0 or 1, for example,
   or an invalid enum discriminant)
 * **Experimental**: Violations of the [Stacked Borrows] rules governing aliasing
   for reference types
 * **Experimental**: Violations of the [Tree Borrows] aliasing rules, as an optional
   alternative to [Stacked Borrows]
-* **Experimental**: Data races
+* **Experimental**: Data races and emulation of weak memory effects, i.e.,
+  atomic reads can return outdated values.
 
 On top of that, Miri will also tell you about memory leaks: when there is memory
 still allocated at the end of the execution, and that memory is not reachable
 from a global `static`, Miri will raise an error.
 
-Miri supports almost all Rust language features; in particular, unwinding and
-concurrency are properly supported (including some experimental emulation of
-weak memory effects, i.e., reads can return outdated values).
-
 You can use Miri to emulate programs on other targets, e.g. to ensure that
 byte-level data manipulation works correctly both on little-endian and
 big-endian systems. See
 [cross-interpretation](#cross-interpretation-running-for-different-targets)
 below.
 
-Miri has already discovered some [real-world bugs](#bugs-found-by-miri). If you
+Miri has already discovered many [real-world bugs](#bugs-found-by-miri). If you
 found a bug with Miri, we'd appreciate if you tell us and we'll add it to the
 list!
 
@@ -45,33 +40,36 @@ clocks, are replaced by deterministic "fake" implementations. Set
 (In particular, the "fake" system RNG APIs make Miri **not suited for
 cryptographic use**! Do not generate keys using Miri.)
 
-All that said, be aware that Miri will **not catch all cases of undefined
-behavior** in your program, and cannot run all programs:
+All that said, be aware that Miri does **not catch every violation of the Rust specification** in
+your program, not least because there is no such specification. Miri uses its own approximation of
+what is and is not Undefined Behavior in Rust. To the best of our knowledge, all Undefined Behavior
+that has the potential to affect a program's correctness *is* being detected by Miri (modulo
+[bugs][I-misses-ub]), but you should consult [the Reference][reference-ub] for the official
+definition of Undefined Behavior. Miri will be updated with the Rust compiler to protect against UB
+as it is understood by the current compiler, but it makes no promises about future versions of
+rustc.
 
-* There are still plenty of open questions around the basic invariants for some
-  types and when these invariants even have to hold. Miri tries to avoid false
-  positives here, so if your program runs fine in Miri right now that is by no
-  means a guarantee that it is UB-free when these questions get answered.
+Further caveats that Miri users should be aware of:
 
-    In particular, Miri does not check that references point to valid data.
 * If the program relies on unspecified details of how data is laid out, it will
   still run fine in Miri -- but might break (including causing UB) on different
-  compiler versions or different platforms.
+  compiler versions or different platforms. (You can use `-Zrandomize-layout`
+  to detect some of these cases.)
 * Program execution is non-deterministic when it depends, for example, on where
   exactly in memory allocations end up, or on the exact interleaving of
   concurrent threads. Miri tests one of many possible executions of your
-  program. You can alleviate this to some extent by running Miri with different
-  values for `-Zmiri-seed`, but that will still by far not explore all possible
-  executions.
+  program, but it will miss bugs that only occur in a different possible execution.
+  You can alleviate this to some extent by running Miri with different
+  values for `-Zmiri-seed`, but that will still by far not explore all possible executions.
 * Miri runs the program as a platform-independent interpreter, so the program
   has no access to most platform-specific APIs or FFI. A few APIs have been
   implemented (such as printing to stdout, accessing environment variables, and
   basic file system access) but most have not: for example, Miri currently does
   not support networking. System API support varies between targets; if you run
   on Windows it is a good idea to use `--target x86_64-unknown-linux-gnu` to get
   better support.
-* Weak memory emulation may [produce weak behaviours](https://github.com/rust-lang/miri/issues/2301)
-  unobservable by compiled programs running on real hardware when `SeqCst` fences are used, and it
+* Weak memory emulation may [produce weak behaviors](https://github.com/rust-lang/miri/issues/2301)
+  when `SeqCst` fences are used that are not actually permitted by the Rust memory model, and it
   cannot produce all behaviors possibly observable on real hardware.
 
 Moreover, Miri fundamentally cannot tell you whether your code is *sound*. [Soundness] is the property
@@ -87,6 +85,8 @@ coverage.
 [Stacked Borrows]: https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md
 [Tree Borrows]: https://perso.crans.org/vanille/treebor/
 [Soundness]: https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#soundness-of-code--of-a-library
+[reference-ub]: https://doc.rust-lang.org/reference/behavior-considered-undefined.html
+[I-misses-ub]: https://github.com/rust-lang/miri/labels/I-misses-UB
 
 
 ## Using Miri
@@ -97,14 +97,8 @@ Install Miri on Rust nightly via `rustup`:
 rustup +nightly component add miri
 ```
 
-If `rustup` says the `miri` component is unavailable, that's because not all
-nightly releases come with all tools. Check out
-[this website](https://rust-lang.github.io/rustup-components-history) to
-determine a nightly version that comes with Miri and install that using `rustup
-toolchain install nightly-YYYY-MM-DD`. Either way, all of the following commands
-assume the right toolchain is pinned via `rustup override set nightly` or
-`rustup override set nightly-YYYY-MM-DD`. (Alternatively, use `cargo
-+nightly`/`cargo +nightly-YYYY-MM-DD` for each of the following commands.)
+All the following commands assume the nightly toolchain is pinned via `rustup override set nightly`.
+Alternatively, use `cargo +nightly` for each of the following commands.
 
 Now you can run your project in Miri:
 
@@ -118,12 +112,12 @@ dependencies. It will ask you for confirmation before installing anything.
 example, `cargo miri test filter` only runs the tests containing `filter` in
 their name.
 
-You can pass arguments to Miri via `MIRIFLAGS`. For example,
+You can pass [flags][miri-flags] to Miri via `MIRIFLAGS`. For example,
 `MIRIFLAGS="-Zmiri-disable-stacked-borrows" cargo miri run` runs the program
 without checking the aliasing of references.
 
 When compiling code via `cargo miri`, the `cfg(miri)` config flag is set for code
-that will be interpret under Miri. You can use this to ignore test cases that fail
+that will be interpreted under Miri. You can use this to ignore test cases that fail
 under Miri because they do things Miri does not support:
 
 ```rust
@@ -159,10 +153,8 @@ endian-sensitive code.
 
 ### Running Miri on CI
 
-To run Miri on CI, make sure that you handle the case where the latest nightly
-does not ship the Miri component because it currently does not build. `rustup
-toolchain install --component` knows how to handle this situation, so the
-following snippet should always work:
+When running Miri on CI, use the following snippet to install a nightly toolchain with the Miri
+component:
 
 ```sh
 rustup toolchain install nightly --component miri
@@ -227,7 +219,7 @@ degree documented below):
 - We have unofficial support (not maintained by the Miri team itself) for some further operating systems.
   - `freebsd`: **maintainer wanted**. Supports `std::env` and parts of `std::{thread, fs}`, but not `std::sync`.
   - `android`: **maintainer wanted**. Support very incomplete, but a basic "hello world" works.
-  - `illumos`: maintained by @devnexen. Support very incomplete, but a basic "hello world" works.
+  - `solaris` / `illumos`: maintained by @devnexen. Support very incomplete, but a basic "hello world" works.
   - `wasm`: **maintainer wanted**. Support very incomplete, not even standard output works, but an empty `main` function works.
 - For targets on other operating systems, Miri might fail before even reaching the `main` function.
 
@@ -273,25 +265,12 @@ To get a backtrace, you need to disable isolation
 RUST_BACKTRACE=1 MIRIFLAGS="-Zmiri-disable-isolation" cargo miri test
 ```
 
-#### "found possibly newer version of crate `std` which `<dependency>` depends on"
-
-Your build directory may contain artifacts from an earlier build that have/have
-not been built for Miri. Run `cargo clean` before switching from non-Miri to
-Miri builds and vice-versa.
-
 #### "found crate `std` compiled by an incompatible version of rustc"
 
 You may be running `cargo miri` with a different compiler version than the one
 used to build the custom libstd that Miri uses, and Miri failed to detect that.
 Try running `cargo miri clean`.
 
-#### "no mir for `std::rt::lang_start_internal`"
-
-This means the sysroot you are using was not compiled with Miri in mind.  This
-should never happen when you use `cargo miri` because that takes care of setting
-up the sysroot.  If you are using `miri` (the Miri driver) directly, see the
-[contributors' guide](CONTRIBUTING.md) for how to use `./miri` to best do that.
-
 
 ## Miri `-Z` flags and environment variables
 [miri-flags]: #miri--z-flags-and-environment-variables
@@ -395,17 +374,17 @@ to Miri failing to detect cases of undefined behavior in a program.
   this flag is **unsound**.
 * `-Zmiri-disable-weak-memory-emulation` disables the emulation of some C++11 weak
   memory effects.
-* `-Zmiri-extern-so-file=<path to a shared object file>` is an experimental flag for providing support
-  for FFI calls. Functions not provided by that file are still executed via the usual Miri shims.
-  **WARNING**: If an invalid/incorrect `.so` file is specified, this can cause undefined behaviour in Miri itself!
-  And of course, Miri cannot do any checks on the actions taken by the external code.
+* `-Zmiri-native-lib=<path to a shared object file>` is an experimental flag for providing support
+  for calling native functions from inside the interpreter via FFI. Functions not provided by that
+  file are still executed via the usual Miri shims.
+  **WARNING**: If an invalid/incorrect `.so` file is specified, this can cause Undefined Behavior in Miri itself!
+  And of course, Miri cannot do any checks on the actions taken by the native code.
   Note that Miri has its own handling of file descriptors, so if you want to replace *some* functions
   working on file descriptors, you will have to replace *all* of them, or the two kinds of
   file descriptors will be mixed up.
   This is **work in progress**; currently, only integer arguments and return values are
   supported (and no, pointer/integer casts to work around this limitation will not work;
-  they will fail horribly). It also only works on unix hosts for now.
-  Follow [the discussion on supporting other types](https://github.com/rust-lang/miri/issues/2365).
+  they will fail horribly). It also only works on Linux hosts for now.
 * `-Zmiri-measureme=<name>` enables `measureme` profiling for the interpreted program.
    This can be used to find which parts of your program are executing slowly under Miri.
    The profile is written out to a file inside a directory called `<name>`, and can be processed
@@ -484,50 +463,14 @@ by all intended entry points, i.e. `cargo miri` and `./miri {test,run}`):
 * `MIRI_SYSROOT` indicates the sysroot to use. When using `cargo miri`, this skips the automatic
   setup -- only set this if you do not want to use the automatically created sysroot. When invoking
   `cargo miri setup`, this indicates where the sysroot will be put.
-* `MIRI_TEST_TARGET` (recognized by `./miri {test,run}`) indicates which target
-  architecture to test against.  `miri` and `cargo miri` accept the `--target` flag for the same
-  purpose.
 * `MIRI_TEST_THREADS` (recognized by `./miri test`): set the number of threads to use for running tests.
   By default, the number of cores is used.
 * `MIRI_NO_STD` makes sure that the target's sysroot is built without libstd. This allows testing
   and running no_std programs. (Miri has a heuristic to detect no-std targets based on the target
   name; this environment variable is only needed when that heuristic fails.)
-* `RUSTC_BLESS` (recognized by `./miri test` and `cargo-miri-test/run-test.py`): overwrite all
-  `stderr` and `stdout` files instead of checking whether the output matches.
 * `MIRI_SKIP_UI_CHECKS` (recognized by `./miri test`): don't check whether the
   `stderr` or `stdout` files match the actual output.
 
-The following environment variables are *internal* and must not be used by
-anyone but Miri itself. They are used to communicate between different Miri
-binaries, and as such worth documenting:
-
-* `MIRI_BE_RUSTC` can be set to `host` or `target`. It tells the Miri driver to
-  actually not interpret the code but compile it like rustc would. With `target`, Miri sets
-  some compiler flags to prepare the code for interpretation; with `host`, this is not done.
-  This environment variable is useful to be sure that the compiled `rlib`s are compatible
-  with Miri.
-* `MIRI_CALLED_FROM_SETUP` is set during the Miri sysroot build,
-  which will re-invoke `cargo-miri` as the `rustc` to use for this build.
-* `MIRI_CALLED_FROM_RUSTDOC` when set to any value tells `cargo-miri` that it is
-  running as a child process of `rustdoc`, which invokes it twice for each doc-test
-  and requires special treatment, most notably a check-only build before interpretation.
-  This is set by `cargo-miri` itself when running as a `rustdoc`-wrapper.
-* `MIRI_CWD` when set to any value tells the Miri driver to change to the given
-  directory after loading all the source files, but before commencing
-  interpretation. This is useful if the interpreted program wants a different
-  working directory at run-time than at build-time.
-* `MIRI_LOCAL_CRATES` is set by `cargo-miri` to tell the Miri driver which
-  crates should be given special treatment in diagnostics, in addition to the
-  crate currently being compiled.
-* `MIRI_ORIG_RUSTDOC` is set and read by different phases of `cargo-miri` to remember the
-  value of `RUSTDOC` from before it was overwritten.
-* `MIRI_REPLACE_LIBRS_IF_NOT_TEST` when set to any value enables a hack that helps bootstrap
-  run the standard library tests in Miri.
-* `MIRI_VERBOSE` when set to any value tells the various `cargo-miri` phases to
-  perform verbose logging.
-* `MIRI_HOST_SYSROOT` is set by bootstrap to tell `cargo-miri` which sysroot to use for *host*
-  operations.
-
 [testing-miri]: CONTRIBUTING.md#testing-the-miri-driver
 
 ## Miri `extern` functions