|
| 1 | +# Using External Repositories |
| 2 | + |
| 3 | +The `rust-lang/rust` git repository depends on several other repos in the `rust-lang` organization. |
| 4 | +There are three main ways we use dependencies: |
| 5 | +1. As a Cargo dependency through crates.io (e.g. `rustc-rayon`) |
| 6 | +2. As a git subtree (e.g. `clippy`) |
| 7 | +3. As a git submodule (e.g. `cargo`) |
| 8 | + |
| 9 | +As a general rule, use crates.io for libraries that could be useful for others in the ecosystem; use |
| 10 | +subtrees for tools that depend on compiler internals and need to be updated if there are breaking |
| 11 | +changes; and use submodules for tools that are independent of the compiler. |
| 12 | + |
| 13 | +## External Dependencies (subtree) |
| 14 | + |
| 15 | +As a developer to this repository, you don't have to treat the following external projects |
| 16 | +differently from other crates that are directly in this repo: |
| 17 | + |
| 18 | +* [Clippy](https://github.com/rust-lang/rust-clippy) |
| 19 | +* [Miri] |
| 20 | +* [rustfmt](https://github.com/rust-lang/rustfmt) |
| 21 | +* [rust-analyzer](https://github.com/rust-lang/rust-analyzer) |
| 22 | + |
| 23 | +In contrast to `submodule` dependencies |
| 24 | +(see below for those), the `subtree` dependencies are just regular files and directories which can |
| 25 | +be updated in tree. However, if possible, enhancements, bug fixes, etc. specific |
| 26 | +to these tools should be filed against the tools directly in their respective |
| 27 | +upstream repositories. The exception is that when rustc changes are required to |
| 28 | +implement a new tool feature or test, that should happen in one collective rustc PR. |
| 29 | + |
| 30 | +### Synchronizing a subtree |
| 31 | + |
| 32 | +Periodically the changes made to subtree based dependencies need to be synchronized between this |
| 33 | +repository and the upstream tool repositories. |
| 34 | + |
| 35 | +Subtree synchronizations are typically handled by the respective tool maintainers. Other users |
| 36 | +are welcome to submit synchronization PRs, however, in order to do so you will need to modify |
| 37 | +your local git installation and follow a very precise set of instructions. |
| 38 | +These instructions are documented, along with several useful tips and tricks, in the |
| 39 | +[syncing subtree changes][clippy-sync-docs] section in Clippy's Contributing guide. |
| 40 | +The instructions are applicable for use with any subtree based tool, just be sure to |
| 41 | +use the correct corresponding subtree directory and remote repository. |
| 42 | + |
| 43 | +The synchronization process goes in two directions: `subtree push` and `subtree pull`. |
| 44 | + |
| 45 | +A `subtree push` takes all the changes that happened to the copy in this repo and creates commits |
| 46 | +on the remote repo that match the local changes. Every local |
| 47 | +commit that touched the subtree causes a commit on the remote repo, but |
| 48 | +is modified to move the files from the specified directory to the tool repo root. |
| 49 | + |
| 50 | +A `subtree pull` takes all changes since the last `subtree pull` |
| 51 | +from the tool repo and adds these commits to the rustc repo along with a merge commit that moves |
| 52 | +the tool changes into the specified directory in the Rust repository. |
| 53 | + |
| 54 | +It is recommended that you always do a push first and get that merged to the tool master branch. |
| 55 | +Then, when you do a pull, the merge works without conflicts. |
| 56 | +While it's definitely possible to resolve conflicts during a pull, you may have to redo the conflict |
| 57 | +resolution if your PR doesn't get merged fast enough and there are new conflicts. Do not try to |
| 58 | +rebase the result of a `git subtree pull`, rebasing merge commits is a bad idea in general. |
| 59 | + |
| 60 | +You always need to specify the `-P` prefix to the subtree directory and the corresponding remote |
| 61 | +repository. If you specify the wrong directory or repository |
| 62 | +you'll get very fun merges that try to push the wrong directory to the wrong remote repository. |
| 63 | +Luckily you can just abort this without any consequences by throwing away either the pulled commits |
| 64 | +in rustc or the pushed branch on the remote and try again. It is usually fairly obvious |
| 65 | +that this is happening because you suddenly get thousands of commits that want to be synchronized. |
| 66 | + |
| 67 | +[clippy-sync-docs]: https://doc.rust-lang.org/nightly/clippy/development/infrastructure/sync.html |
| 68 | + |
| 69 | +### Creating a new subtree dependency |
| 70 | + |
| 71 | +If you want to create a new subtree dependency from an existing repository, call (from this |
| 72 | +repository's root directory!) |
| 73 | + |
| 74 | +``` |
| 75 | +git subtree add -P src/tools/clippy https://github.com/rust-lang/rust-clippy.git master |
| 76 | +``` |
| 77 | + |
| 78 | +This will create a new commit, which you may not rebase under any circumstances! Delete the commit |
| 79 | +and redo the operation if you need to rebase. |
| 80 | + |
| 81 | +Now you're done, the `src/tools/clippy` directory behaves as if Clippy were |
| 82 | +part of the rustc monorepo, so no one but you (or others that synchronize |
| 83 | +subtrees) actually needs to use `git subtree`. |
| 84 | + |
| 85 | + |
| 86 | +## External Dependencies (submodules) |
| 87 | + |
| 88 | +Building Rust will also use external git repositories tracked using [git |
| 89 | +submodules]. The complete list may be found in the [`.gitmodules`] file. Some |
| 90 | +of these projects are required (like `stdarch` for the standard library) and |
| 91 | +some of them are optional (like [Miri]). |
| 92 | + |
| 93 | +Usage of submodules is discussed more in the [Using Git |
| 94 | +chapter](git.md#git-submodules). |
| 95 | + |
| 96 | +Some of the submodules are allowed to be in a "broken" state where they |
| 97 | +either don't build or their tests don't pass, e.g. the documentation books |
| 98 | +like [The Rust Reference]. Maintainers of these projects will be notified |
| 99 | +when the project is in a broken state, and they should fix them as soon |
| 100 | +as possible. The current status is tracked on the [toolstate website]. |
| 101 | +More information may be found on the Forge [Toolstate chapter]. |
| 102 | + |
| 103 | +Breakage is not allowed in the beta and stable channels, and must be addressed |
| 104 | +before the PR is merged. They are also not allowed to be broken on master in |
| 105 | +the week leading up to the beta cut. |
| 106 | + |
| 107 | +[git submodules]: https://git-scm.com/book/en/v2/Git-Tools-Submodules |
| 108 | +[`.gitmodules`]: https://github.com/rust-lang/rust/blob/master/.gitmodules |
| 109 | +[The Rust Reference]: https://github.com/rust-lang/reference/ |
| 110 | +[toolstate website]: https://rust-lang-nursery.github.io/rust-toolstate/ |
| 111 | +[Toolstate chapter]: https://forge.rust-lang.org/infra/toolstate.html |
| 112 | + |
| 113 | +### Breaking Tools Built With The Compiler |
| 114 | + |
| 115 | +Rust's build system builds a number of tools that make use of the internals of |
| 116 | +the compiler and that are hosted in a separate repository, and included in Rust |
| 117 | +via git submodules (such as [Miri]). If these tools break because of your |
| 118 | +changes, you may run into a sort of "chicken and egg" problem. These tools rely |
| 119 | +on the latest compiler to be built so you can't update them (in their own |
| 120 | +repositories) to reflect your changes to the compiler until those changes are |
| 121 | +merged into the compiler. At the same time, you can't get your changes merged |
| 122 | +into the compiler because the rust-lang/rust build won't pass until those tools |
| 123 | +build and pass their tests. |
| 124 | + |
| 125 | +Luckily, a feature was |
| 126 | +[added to Rust's build](https://github.com/rust-lang/rust/issues/45861) to make |
| 127 | +all of this easy to handle. The idea is that we allow these tools to be |
| 128 | +"broken", so that the rust-lang/rust build passes without trying to build them, |
| 129 | +then land the change in the compiler, and go update the tools that you |
| 130 | +broke. Some tools will require waiting for a nightly release before this can |
| 131 | +happen, while others use the builds uploaded after each bors merge and thus can |
| 132 | +be updated immediately (check the tool's documentation for details). Once you're |
| 133 | +done and the tools are working again, you go back in the compiler and update the |
| 134 | +tools so they can be distributed again. |
| 135 | + |
| 136 | +This should avoid a bunch of synchronization dances and is also much easier on contributors as |
| 137 | +there's no need to block on tools changes going upstream. |
| 138 | + |
| 139 | +Here are those same steps in detail: |
| 140 | + |
| 141 | +1. (optional) First, if it doesn't exist already, create a `config.toml` by copying |
| 142 | + `config.example.toml` in the root directory of the Rust repository. |
| 143 | + Set `submodules = false` in the `[build]` section. This will prevent `x.py` |
| 144 | + from resetting to the original branch after you make your changes. If you |
| 145 | + need to [update any submodules to their latest versions](#updating-submodules), |
| 146 | + see the section of this file about that for more information. |
| 147 | +2. (optional) Run `./x.py test src/tools/cargo` (substituting the submodule |
| 148 | + that broke for `cargo`). Fix any errors in the submodule (and possibly others). |
| 149 | +3. (optional) Make commits for your changes and send them to upstream repositories as a PR. |
| 150 | +4. (optional) Maintainers of these submodules will **not** merge the PR. The PR can't be |
| 151 | + merged because CI will be broken. You'll want to write a message on the PR referencing |
| 152 | + your change, and how the PR should be merged once your change makes it into a nightly. |
| 153 | +5. Wait for your PR to merge. |
| 154 | +6. Wait for a nightly. |
| 155 | +7. (optional) Help land your PR on the upstream repository now that your changes are in nightly. |
| 156 | +8. (optional) Send a PR to rust-lang/rust updating the submodule. |
| 157 | + |
0 commit comments