Skip to content

Do not get proc_macro from the sysroot in rustc #141595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 29, 2025
Merged

Do not get proc_macro from the sysroot in rustc #141595

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions Cargo.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3444,6 +3444,7 @@ dependencies = [
"rustc_macros",
"rustc_parse",
"rustc_parse_format",
"rustc_proc_macro",
"rustc_session",
"rustc_span",
"rustc_target",
Expand Down Expand Up @@ -3733,6 +3734,7 @@ dependencies = [
"rustc_lint_defs",
"rustc_macros",
"rustc_parse",
"rustc_proc_macro",
"rustc_serialize",
"rustc_session",
"rustc_span",
Expand Down Expand Up @@ -4081,6 +4083,7 @@ dependencies = [
"rustc_index",
"rustc_macros",
"rustc_middle",
"rustc_proc_macro",
"rustc_serialize",
"rustc_session",
"rustc_span",
Expand Down Expand Up @@ -4337,6 +4340,13 @@ dependencies = [
"tracing",
]

[[package]]
name = "rustc_proc_macro"
version = "0.0.0"
dependencies = [
"rustc-literal-escaper",
]

[[package]]
name = "rustc_query_impl"
version = "0.0.0"
Expand Down
3 changes: 3 additions & 0 deletions compiler/rustc_builtin_macros/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ rustc_lint_defs = { path = "../rustc_lint_defs" }
rustc_macros = { path = "../rustc_macros" }
rustc_parse = { path = "../rustc_parse" }
rustc_parse_format = { path = "../rustc_parse_format" }
# We must use the proc_macro version that we will compile proc-macros against,
# not the one from our own sysroot.
rustc_proc_macro = { path = "../rustc_proc_macro" }
rustc_session = { path = "../rustc_session" }
rustc_span = { path = "../rustc_span" }
rustc_target = { path = "../rustc_target" }
Expand Down
4 changes: 1 addition & 3 deletions compiler/rustc_builtin_macros/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@
#![recursion_limit = "256"]
// tidy-alphabetical-end

extern crate proc_macro;

use std::sync::Arc;

use rustc_expand::base::{MacroExpanderFn, ResolverExpand, SyntaxExtensionKind};
Expand Down Expand Up @@ -139,7 +137,7 @@ pub fn register_builtin_macros(resolver: &mut dyn ResolverExpand) {
CoercePointee: coerce_pointee::expand_deriving_coerce_pointee,
}

let client = proc_macro::bridge::client::Client::expand1(proc_macro::quote);
let client = rustc_proc_macro::bridge::client::Client::expand1(rustc_proc_macro::quote);
register(sym::quote, SyntaxExtensionKind::Bang(Arc::new(BangProcMacro { client })));
let requires = SyntaxExtensionKind::Attr(Arc::new(contracts::ExpandRequires));
register(sym::contracts_requires, requires);
Expand Down
3 changes: 3 additions & 0 deletions compiler/rustc_expand/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ rustc_lexer = { path = "../rustc_lexer" }
rustc_lint_defs = { path = "../rustc_lint_defs" }
rustc_macros = { path = "../rustc_macros" }
rustc_parse = { path = "../rustc_parse" }
# We must use the proc_macro version that we will compile proc-macros against,
# not the one from our own sysroot.
rustc_proc_macro = { path = "../rustc_proc_macro" }
rustc_serialize = { path = "../rustc_serialize" }
rustc_session = { path = "../rustc_session" }
rustc_span = { path = "../rustc_span" }
Expand Down
2 changes: 0 additions & 2 deletions compiler/rustc_expand/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@
#![feature(yeet_expr)]
// tidy-alphabetical-end

extern crate proc_macro as pm;

mod build;
mod errors;
// FIXME(Nilstrieb) Translate macro_rules diagnostics
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_expand/src/proc_macro.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
use rustc_ast as ast;
use rustc_ast::ptr::P;
use rustc_ast::tokenstream::TokenStream;
use rustc_errors::ErrorGuaranteed;
use rustc_parse::parser::{ForceCollect, Parser};
use rustc_session::config::ProcMacroExecutionStrategy;
use rustc_span::Span;
use rustc_span::profiling::SpannedEventArgRecorder;
use {rustc_ast as ast, rustc_proc_macro as pm};

use crate::base::{self, *};
use crate::{errors, proc_macro_server};
Expand Down
16 changes: 8 additions & 8 deletions compiler/rustc_expand/src/proc_macro_server.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
use std::ops::{Bound, Range};

use ast::token::IdentIsRaw;
use pm::bridge::{
DelimSpan, Diagnostic, ExpnGlobals, Group, Ident, LitKind, Literal, Punct, TokenTree, server,
};
use pm::{Delimiter, Level};
use rustc_ast as ast;
use rustc_ast::token;
use rustc_ast::tokenstream::{self, DelimSpacing, Spacing, TokenStream};
Expand All @@ -15,6 +11,10 @@ use rustc_errors::{Diag, ErrorGuaranteed, MultiSpan, PResult};
use rustc_parse::lexer::nfc_normalize;
use rustc_parse::parser::Parser;
use rustc_parse::{exp, new_parser_from_source_str, source_str_to_stream, unwrap_or_emit_fatal};
use rustc_proc_macro::bridge::{
DelimSpan, Diagnostic, ExpnGlobals, Group, Ident, LitKind, Literal, Punct, TokenTree, server,
};
use rustc_proc_macro::{Delimiter, Level};
use rustc_session::parse::ParseSess;
use rustc_span::def_id::CrateNum;
use rustc_span::{BytePos, FileName, Pos, Span, Symbol, sym};
Expand Down Expand Up @@ -66,7 +66,7 @@ impl FromInternal<token::LitKind> for LitKind {
token::CStr => LitKind::CStr,
token::CStrRaw(n) => LitKind::CStrRaw(n),
token::Err(_guar) => {
// This is the only place a `pm::bridge::LitKind::ErrWithGuar`
// This is the only place a `rustc_proc_macro::bridge::LitKind::ErrWithGuar`
// is constructed. Note that an `ErrorGuaranteed` is available,
// as required. See the comment in `to_internal`.
LitKind::ErrWithGuar
Expand Down Expand Up @@ -149,7 +149,7 @@ impl FromInternal<(TokenStream, &mut Rustc<'_, '_>)> for Vec<TokenTree<TokenStre
}

trees.push(TokenTree::Group(Group {
delimiter: pm::Delimiter::from_internal(delim),
delimiter: rustc_proc_macro::Delimiter::from_internal(delim),
stream: Some(stream),
span: DelimSpan {
open: span.open,
Expand Down Expand Up @@ -270,7 +270,7 @@ impl FromInternal<(TokenStream, &mut Rustc<'_, '_>)> for Vec<TokenTree<TokenStre
let stream =
TokenStream::token_alone(token::Lifetime(ident.name, is_raw), ident.span);
trees.push(TokenTree::Group(Group {
delimiter: pm::Delimiter::None,
delimiter: rustc_proc_macro::Delimiter::None,
stream: Some(stream),
span: DelimSpan::from_single(span),
}))
Expand Down Expand Up @@ -302,7 +302,7 @@ impl FromInternal<(TokenStream, &mut Rustc<'_, '_>)> for Vec<TokenTree<TokenStre
trees.push(TokenTree::Punct(Punct { ch: b'!', joint: false, span }));
}
trees.push(TokenTree::Group(Group {
delimiter: pm::Delimiter::Bracket,
delimiter: rustc_proc_macro::Delimiter::Bracket,
stream: Some(stream),
span: DelimSpan::from_single(span),
}));
Expand Down
3 changes: 3 additions & 0 deletions compiler/rustc_metadata/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ rustc_hir_pretty = { path = "../rustc_hir_pretty" }
rustc_index = { path = "../rustc_index" }
rustc_macros = { path = "../rustc_macros" }
rustc_middle = { path = "../rustc_middle" }
# We must use the proc_macro version that we will compile proc-macros against,
# not the one from our own sysroot.
rustc_proc_macro = { path = "../rustc_proc_macro" }
rustc_serialize = { path = "../rustc_serialize" }
rustc_session = { path = "../rustc_session" }
rustc_span = { path = "../rustc_span" }
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_metadata/src/creader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ use std::str::FromStr;
use std::time::Duration;
use std::{cmp, env, iter};

use proc_macro::bridge::client::ProcMacro;
use rustc_ast::expand::allocator::{AllocatorKind, alloc_error_handler_name, global_fn_name};
use rustc_ast::{self as ast, *};
use rustc_data_structures::fx::FxHashSet;
Expand All @@ -23,6 +22,7 @@ use rustc_hir::definitions::Definitions;
use rustc_index::IndexVec;
use rustc_middle::bug;
use rustc_middle::ty::{TyCtxt, TyCtxtFeed};
use rustc_proc_macro::bridge::client::ProcMacro;
use rustc_session::config::{
self, CrateType, ExtendedTargetModifierInfo, ExternLocation, OptionsTargetModifiers,
TargetModifier,
Expand Down
2 changes: 0 additions & 2 deletions compiler/rustc_metadata/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@
#![feature(trusted_len)]
// tidy-alphabetical-end

extern crate proc_macro;

pub use rmeta::provide;

mod dependency_format;
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_metadata/src/rmeta/decoder.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ use std::sync::{Arc, OnceLock};
use std::{io, iter, mem};

pub(super) use cstore_impl::provide;
use proc_macro::bridge::client::ProcMacro;
use rustc_ast as ast;
use rustc_data_structures::fingerprint::Fingerprint;
use rustc_data_structures::fx::FxIndexMap;
Expand All @@ -26,6 +25,7 @@ use rustc_middle::mir::interpret::{AllocDecodingSession, AllocDecodingState};
use rustc_middle::ty::Visibility;
use rustc_middle::ty::codec::TyDecoder;
use rustc_middle::{bug, implement_ty_decoder};
use rustc_proc_macro::bridge::client::ProcMacro;
use rustc_serialize::opaque::MemDecoder;
use rustc_serialize::{Decodable, Decoder};
use rustc_session::Session;
Expand Down
21 changes: 21 additions & 0 deletions compiler/rustc_proc_macro/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# We need to use a separate crate including library/proc_macro as opposed to a
# direct path dependency on library/proc_macro because doing the latter will
# cause two copies of libproc_macro.rlib to end up in the sysroot, breaking
# proc-macro crates. In addition it confuses the workspace_members function of
# bootstrap.

[package]
name = "rustc_proc_macro"
version = "0.0.0"
edition = "2024"

[lib]
path = "../../library/proc_macro/src/lib.rs"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will break for the same reason, no? That path just doesn't exist when rustc-dev is distributed via rustup.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explicitly added it to the rustc-dev component. See #141595 (comment).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm... if you do that then you can go back to making it a regular path dependency, you don't need this separate crate, do you?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That didn't work. Building two crates both named proc_macro (once as part of stdlib and once as part of rustc) causes ambiguity for extern crate proc_macro; when the rustc libraries are part of the sysroot. Because of this I'm forced to build the rustc copy with a different crate name, which requires a separate Cargo.toml.

test = false
doctest = false

[dependencies]
rustc-literal-escaper = "0.0.2"

[features]
rustc-dep-of-std = []
4 changes: 4 additions & 0 deletions library/proc_macro/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,7 @@ std = { path = "../std" }
# loaded from sysroot causing duplicate lang items and other similar errors.
core = { path = "../core" }
rustc-literal-escaper = { version = "0.0.2", features = ["rustc-dep-of-std"] }

[features]
default = ["rustc-dep-of-std"]
rustc-dep-of-std = []
3 changes: 2 additions & 1 deletion library/proc_macro/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
#![recursion_limit = "256"]
#![allow(internal_features)]
#![deny(ffi_unwind_calls)]
#![allow(rustc::internal)] // Can't use FxHashMap when compiled as part of the standard library
#![warn(rustdoc::unescaped_backticks)]
#![warn(unreachable_pub)]
#![deny(unsafe_op_in_unsafe_fn)]
Expand Down Expand Up @@ -95,7 +96,7 @@ pub fn is_available() -> bool {
///
/// This is both the input and output of `#[proc_macro]`, `#[proc_macro_attribute]`
/// and `#[proc_macro_derive]` definitions.
#[rustc_diagnostic_item = "TokenStream"]
#[cfg_attr(feature = "rustc-dep-of-std", rustc_diagnostic_item = "TokenStream")]
#[stable(feature = "proc_macro_lib", since = "1.15.0")]
#[derive(Clone)]
pub struct TokenStream(Option<bridge::client::TokenStream>);
Expand Down
3 changes: 2 additions & 1 deletion src/bootstrap/src/core/build_steps/dist.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -776,7 +776,8 @@ impl Step for RustcDev {
copy_src_dirs(
builder,
&builder.src,
&["compiler"],
// The compiler has a path dependency on proc_macro, so make sure to include it.
&["compiler", "library/proc_macro"],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested that cargo metadata and cargo check still work in the built rustc-dev tarball. It may be possible to do the same trick for rustc-literal-escaper to pull it back into this repo.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rustup component was not the only problem; the other problem are out-of-tree consumers -- RA has to be able to import some of the rustc crates, and IIRC the literal-escaper occurred in the dependency tree.

It'd be good to involve them here as well in case rustc_builtin_macros is in their dependency tree -- Cc @Veykril

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not so that should be fine

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am also iffy about the fact that the compiler tree relies on this somewhat magical hack, but it looks fairly simple/maintainable and works perfectly fine. It seems like no one in the project is bothered by it (yet), so I would say let's move forward and revisit it if someone raises an issue.

&[],
&tarball.image_dir().join("lib/rustlib/rustc-src/rust"),
);
Expand Down
Loading