Macro protocols lack async from expansion function requirements despite proposals stating they should have it

[roopekv]

Issue

Swift's concurrency features cannot be fully utilized in macros without being able to implement expansion function requirements of macro protocols as async functions.

Cause

While the Swift Evolution proposals for expression and attached macros state that the expansion function requirements of macro protocols should be async, the macro protocol declarations in swift-syntax are missing the async keyword from their expansion function requirements (see comparison below), making it impossible to implement them as async functions.

Solution

Changing the declarations in swift-syntax to async shouldn't break existing macros, as async function requirements can be implemented as non-async functions.

Example

protocol Macro {
    func expansion()
}

struct SyncMacro : Macro {
    func expansion() {} // OK ✅
}

struct AsyncMacro : Macro {
    func expansion() async {} // ERROR ❌
}

protocol Macro {
    func expansion() async
}

struct SyncMacro : Macro {
    func expansion() {} // OK ✅
}

struct AsyncMacro : Macro {
    func expansion() async {} // OK ✅
}

I acknowledge that additional changes would have to be made within the swift-syntax package (changes at the expansion function call sites and possible further structural changes to facilitate asynchronous code), but depending on how they are implemented, the changes don't have to affect the user-facing parts of the package in a source breaking way.

Differences between Swift Evolution proposals and swift-syntax

ExpressionMacro

Proposal

public protocol ExpressionMacro: FreestandingMacro {
  /// Expand a macro described by the given freestanding macro expansion
  /// within the given context to produce a replacement expression.
  static func expansion(
    of node: some FreestandingMacroExpansionSyntax,
    in context: some MacroExpansionContext
  ) async throws -> ExprSyntax
}

swift-syntax

swift-syntax/Sources/SwiftSyntaxMacros/MacroProtocols/ExpressionMacro.swift

Lines 20 to 27 in 0b324f8

	public protocol ExpressionMacro: FreestandingMacro {
	/// Expand a macro described by the given freestanding macro expansion
	/// within the given context to produce a replacement expression.
	static func expansion(
	of node: some FreestandingMacroExpansionSyntax,
	in context: some MacroExpansionContext
	) throws -> ExprSyntax
	}

PeerMacro

Proposal

public PeerMacro: AttachedMacro {
  /// Expand a macro described by the given attribute to
  /// produce "peer" declarations of the declaration to which it
  /// is attached.
  ///
  /// The macro expansion can introduce "peer" declarations that 
  /// go alongside the given declaration.
  static func expansion(
    of node: AttributeSyntax,
    providingPeersOf declaration: some DeclSyntaxProtocol,
    in context: some MacroExpansionContext
  ) async throws -> [DeclSyntax]
}

swift-syntax

swift-syntax/Sources/SwiftSyntaxMacros/MacroProtocols/PeerMacro.swift

Lines 17 to 29 in 0b324f8

	public protocol PeerMacro: AttachedMacro {
	/// Expand a macro described by the given custom attribute and
	/// attached to the given declaration and evaluated within a
	/// particular expansion context.
	///
	/// The macro expansion can introduce "peer" declarations that sit alongside
	/// the given declaration.
	static func expansion(
	of node: AttributeSyntax,
	providingPeersOf declaration: some DeclSyntaxProtocol,
	in context: some MacroExpansionContext
	) throws -> [DeclSyntax]
	}

MemberMacro

Proposal

protocol MemberMacro: AttachedMacro {
  /// Expand a macro described by the given attribute to
  /// produce additional members of the given declaration to which
  /// the attribute is attached.
  static func expansion(
    of node: AttributeSyntax,
    providingMembersOf declaration: some DeclGroupSyntax,
    in context: some MacroExpansionContext
  ) async throws -> [DeclSyntax]
}

swift-syntax

swift-syntax/Sources/SwiftSyntaxMacros/MacroProtocols/MemberMacro.swift

Lines 20 to 71 in 0b324f8

	public protocol MemberMacro: AttachedMacro {
	/// Expand an attached declaration macro to produce a set of members.
	///
	/// - Parameters:
	/// - node: The custom attribute describing the attached macro.
	/// - declaration: The declaration the macro attribute is attached to.
	/// - context: The context in which to perform the macro expansion.
	///
	/// - Returns: the set of member declarations introduced by this macro, which
	/// are nested inside the `attachedTo` declaration.
	///
	/// - Warning: This is the legacy `expansion` function of `MemberMacro` that is provided for backwards-compatiblity.
	/// Use ``expansion(of:providingMembersOf:conformingTo:in:)-1sxoe`` instead.
	static func expansion(
	of node: AttributeSyntax,
	providingMembersOf declaration: some DeclGroupSyntax,
	in context: some MacroExpansionContext
	) throws -> [DeclSyntax]
	/// Expand an attached declaration macro to produce a set of members.
	///
	/// - Parameters:
	/// - node: The custom attribute describing the attached macro.
	/// - declaration: The declaration the macro attribute is attached to.
	/// - conformingTo: The set of protocols that were declared
	/// in the set of conformances for the macro and to which the declaration
	/// does not explicitly conform. The member macro itself cannot declare
	/// conformances to these protocols (only an extension macro can do that),
	/// but can provide supporting declarations, such as a required
	/// initializer or stored property, that cannot be written in an
	/// extension.
	/// - context: The context in which to perform the macro expansion.
	///
	/// - Returns: the set of member declarations introduced by this macro, which
	/// are nested inside the `attachedTo` declaration.
	static func expansion(
	of node: AttributeSyntax,
	providingMembersOf declaration: some DeclGroupSyntax,
	conformingTo protocols: [TypeSyntax],
	in context: some MacroExpansionContext
	) throws -> [DeclSyntax]
	}
	private struct UnimplementedExpansionMethodError: Error, CustomStringConvertible {
	var description: String {
	"""
	Types conforming to `MemberMacro` must implement either \
	expansion(of:providingMembersOf:in:) or \
	expansion(of:providingMembersOf:conformingTo:in:)
	"""
	}
	}

AccessorMacro

Proposal

protocol AccessorMacro: AttachedMacro {
  /// Expand a macro described by the given attribute to
  /// produce accessors for the given declaration to which
  /// the attribute is attached.
  static func expansion(
    of node: AttributeSyntax,
    providingAccessorsOf declaration: some DeclSyntaxProtocol,
    in context: some MacroExpansionContext
  ) async throws -> [AccessorDeclSyntax]
}

swift-syntax

swift-syntax/Sources/SwiftSyntaxMacros/MacroProtocols/AccessorMacro.swift

Lines 18 to 27 in 0b324f8

	public protocol AccessorMacro: AttachedMacro {
	/// Expand a macro that's expressed as a custom attribute attached to
	/// the given declaration. The result is a set of accessors for the
	/// declaration.
	static func expansion(
	of node: AttributeSyntax,
	providingAccessorsOf declaration: some DeclSyntaxProtocol,
	in context: some MacroExpansionContext
	) throws -> [AccessorDeclSyntax]
	}

MemberAttributeMacro

Proposal

protocol MemberAttributeMacro: AttachedMacro {
  /// Expand a macro described by the given custom attribute to
  /// produce additional attributes for the members of the type.
  static func expansion(
    of node: AttributeSyntax,
    attachedTo declaration: some DeclGroupSyntax,
    providingAttributesOf member: some DeclSyntaxProtocol,
    in context: some MacroExpansionContext
  ) async throws -> [AttributeSyntax]
}

swift-syntax

swift-syntax/Sources/SwiftSyntaxMacros/MacroProtocols/MemberAttributeMacro.swift

Lines 21 to 38 in 0b324f8

	public protocol MemberAttributeMacro: AttachedMacro {
	/// Expand an attached declaration macro to produce an attribute list for
	/// a given member.
	///
	/// - Parameters:
	/// - node: The custom attribute describing the attached macro.
	/// - declaration: The declaration the macro attribute is attached to.
	/// - member: The member declaration to attach the resulting attributes to.
	/// - context: The context in which to perform the macro expansion.
	///
	/// - Returns: the set of attributes to apply to the given member.
	static func expansion(
	of node: AttributeSyntax,
	attachedTo declaration: some DeclGroupSyntax,
	providingAttributesFor member: some DeclSyntaxProtocol,
	in context: some MacroExpansionContext
	) throws -> [AttributeSyntax]
	}

Additional quotes from Expression Macros proposal

The macro expansion operation is asynchronous, to account for potentially-asynchronous operations that will eventually be added to MacroExpansionContext. For example, operations that require additional communication with the compiler to get types of subexpressions, access files in the program, and so on.

* Make the ExpressionMacro.expansion(of:in:) requirement async.

[ahoppen]

Synced to Apple’s issue tracker as rdar://133763644

[ahoppen]

The swift-evolution proposals for macros were inconsistent about whether expansion should be async or not and the motivation for making the expansion functions async never happened.

Given that swift-syntax has always had synchronous expansion functions, I’m updating the proposals to also state synchronous expansions functions. swiftlang/swift-evolution#2546

[roopekv]

@ahoppen that would be the easiest way to solve this issue, but wouldn't it be better for expansion to be async? While MacroExpansionContext doesn't currently have async operations, being able to use concurrency features inside macros would still be useful.

[ahoppen]

What kind of async operations would you like to do in the macro?

[roopekv]

Use await, and concurrency features in general.

[ahoppen]

What kind of concurrent operations would you like to perform that need to be asynchronous? I’m trying to understand the use cases in which an asynchronous expansion function would help.

[roopekv]

The ability to perform expensive operations in parallel could significantly speed up the expansion of computationally heavy macros.

Being able to wait for completion and results of async functions is also an important feature in itself, especially for cases where a function you need to use has no sync alternatives.

Are there downsides to supporting async expansion? The change wouldn't break existing macros.

[ahoppen]

I don’t think there any fundamental obstacles to providing asynchronous expansion functions. It mostly requires engineering work.

My thinking is that async expansion functions wouldn’t offer huge benefits though because macros can’t do IO or network requests, which are traditionally a good use case for asynchronous functions and operate on sufficiently small inputs that the computation should be sufficiently quick that it wouldn’t benefit from multi-threading. If you have concrete counter-examples I’m happy to hear them and prove my assumptions wrong.

[roopekv]

Whether a macro benefits from multi-threading depends on what it does, not necessarily on the size of the inputs it receives.

Example of a macro that could hugely benefit from multi-threading if expensiveOperation lives up to its name:

@freestanding(expression)
public macro lookupTable(input: [Input]) -> [Input: Output] = #externalMacro(module: "LookupTableMacros", type: "LookupTableMacro")

public struct LookupTableMacro: ExpressionMacro {
    public static func expansion(of node: some FreestandingMacroExpansionSyntax, in context: some MacroExpansionContext) -> ExprSyntax {
        let inputs: [Input] = getInputs(node)
        
        var lookupTable = [String](repeating: String(), count: inputs.count)
        
        for i in 0..<inputs.count {
            let input = inputs[i]
            let output = expensiveOperation(input)
            lookupTable[i] = "\(input): \(output)"
        }
        
        return "[\n\(raw: lookupTable.joined(separator: ",\n"))\n]"
    }
}

let lookupTable = #lookupTable(input: [
    <#Input 1#>,
    <#Input 2#>,
    <#Input 3#>,
    ...
])

/* Expansion:
[
    <#Input 1#>: <#Output 1#>,
    <#Input 2#>: <#Output 2#>,
    <#Input 3#>: <#Output 3#>,
    ...
]*/

The code above depicts a macro that could be used to generate a compile-time lookup table for the specified inputs. If expensiveOperation could be performed for multiple inputs in parallel instead of serially, the speed-ups could be huge.

Let's say that:

• expensiveOperation takes 1 s to perform
• You call #lookupTable with 24 inputs
• Your machine can process 24 threads in parallel

Then the expansion could theoretically take:

• 24 s without multi-threading
• 1 s with multi-threading

Is this example sufficient @ahoppen ?

Also, regarding the engineering work required for providing asynchronous expansion functions: if someone were to implement the feature and make an acceptable pull request, could it simply be accepted into the next release of swift-syntax, or would it require a Swift Evolution proposal or an amendment to the existing ones?

[ahoppen]

I understand how parallel processing helps speed up things. I am just wondering what kind of operation expensiveOperation might come up in practice that takes 1s to run. And to be clear: I agree that having async expansion functions could make sense, I’m just wondering how much real-world benefits they could provide.

I think the most tricky part about this change is considering API compatibility. If expansion becomes async, then all callers of it need to become async and the primary user facing caller would be assertMacroExpansion. This means that users would need to update all their test cases and I think that’s a pretty big API change if the benefits from the async expansion function are mostly theoretical.

[roopekv]

Thanks for bringing up assertMacroExpansion. I hadn't considered how testing works for macros and incorrectly assumed that macros always run as their separate processes in user-facing code.

So yes, to be able to call assertMacroExpansion from any context and guarantee forward progress, assertMacroExpansion and certain internal swift-syntax functions would have to be changed to async, which would be an API-incompatible change for the users of assertMacroExpansion.

However, as assertMacroExpansion is mainly used in tests, and both of the major testing frameworks (XCTest and swift-testing) support asynchronous tests, the code that this change would break could be easily fixed by marking the tests that call assertMacroExpansion as async and adding the await keyword in front of the assertMacroExpansion function calls.

The amount of code this change would break should be relatively small, considering that macros have been around for less than a year at the time of writing. There have been similarly and more drastically API-incompatible changes made to swift-syntax in the past.

The benefits of async expansion functions are not theoretical. Being able to call async functions and wait for their completion and results is a practical feature, crucial for situations where there are no sync alternatives for the function that you need to use in your macro. While parallelism is not a requirement, the speed-ups it can provide for certain macros are a practical benefit. Async expansion functions are also a requirement for adding async operations to MacroExpansionContext, as mentioned in the expression macros proposal:

The macro expansion operation is asynchronous, to account for potentially-asynchronous operations that will eventually be added to MacroExpansionContext. For example, operations that require additional communication with the compiler to get types of subexpressions, access files in the program, and so on.

As the swift-syntax package uses semantic versioning, minor API breakage is acceptable between major releases. Unlike some of the previous API-incompatible changes like renaming a property for clearer meaning, changing assertMacroExpansion to async would have tangible benefits not only for the users of swift-syntax but for the Swift ecosystem as a whole.

[ahoppen]

The amount of code this change would break should be relatively small.

I don’t think it’s a small change. Macros tend to have tests and all of them need to be manually updated, so it’s certainly has a significant impact. It’s certainly not the end of the world but it needs to be factored in.

[roopekv]

@ahoppen I might have found a solution that avoids breaking anything:

protocol Macro {
    func expansion() // must be implemented
    func expansion() async // can be implemented if required
}

// default async expansion is sync expansion
extension Macro {
    func expansion() async {
        let syncExpansion = { expansion() }
        syncExpansion()
    }
}

// calls sync expansion
func assertMacroExpansion(macro: some Macro) {
    macro.expansion()
}

// calls async expansion
func assertMacroExpansion(macro: some Macro) async {
    await macro.expansion()
}

struct SimpleMacro : Macro {
    func expansion() {}
}

struct AsyncOnlyMacro : Macro {
    func expansion() {
        fatalError("must use async assertMacroExpansion")
    }
    
    func expansion() async {}
}

// calling macro always calls async expansion
#simpleMacro() // calls sync expansion through async expansion (default)
#asyncOnlyMacro() // calls async expansion

[roopekv]

The above solution would require some code duplication, but shouldn't break anything. If wanted, sync expansion could be deprecated in a later version and eventually be removed to eliminate the duplicate code.

[ahoppen]

I think this could be an interesting approach for migration. I’m happy to review it. Would you like to flesh this out in a PR?

[roopekv]

Gladly :D

I don't have much experience contributing to open-source projects, but I can try my best to make a PR. Not sure how long it will take, could be days or weeks, but I can start working on it tomorrow.

[ahoppen]

Thank you @roopekv. Let me know if you have any questions. https://github.com/swiftlang/swift-syntax/blob/main/CONTRIBUTING.md should be a good place to start.