[clang] Add support for “regular” keyword attributes

Platform-specific language extensions often want to provide a way of
indicating that certain functions should be called in a different way,
compiled in a different way, or otherwise treated differently from a
“normal” function.  Honoring these indications is often required for
correctness, rather being than an optimization/QoI thing.

If a function declaration has a property P that matters for correctness,
it will be ODR-incompatible with a function that does not have property P.
If a function type has a property P that affects the calling convention,
it will not be two-way compatible with a function type that does not
have property P.  These properties therefore affect language semantics.
That in turn means that they cannot be treated as standard [[]]
attributes.

Until now, many of these properties have been specified using GNU-style
attributes instead.  GNU attributes have traditionally been more lax
than standard attributes, with many of them having semantic meaning.
Examples include calling conventions and the vector_size attribute.

However, there is a big drawback to using GNU attributes for semantic
information: compilers that don't understand the attributes will
(by default) emit a warning rather than an error.  They will go on to
compile the code as though the attributes weren't present, which will
inevitably lead to wrong code in most cases.  For users who live
dangerously and disable the warning, this wrong code could even be
generated silently.

A more robust approach would be to specify the properties using
keywords, which older compilers would then reject.  Some vendor-specific
extensions have already taken this approach.  But traditionally, each
such keyword has been treated as a language extension in its own right.
This has three major drawbacks:

(1) The parsing rules need to be kept up-to-date as the language evolves.

(2) There are often corner cases that similar extensions handle differently.

(3) Each extension requires more custom code than a standard attribute.

The underlying problem for all three is that, unlike for true attributes,
there is no established template that extensions can reuse.  The purpose
of this patch series is to try to provide such a template.

One option would have been to pick an existing keyword and do whatever
that keyword does.  The problem with that is that most keywords only
apply to specific kinds of types, kinds of decls, etc., and so the
parsing rules are (for good reason) not generally applicable to all
types and decls.

Really, the “only” thing wrong with using standard attributes is that
standard attributes cannot affect semantics.  In all other respects
they provide exactly what we need: a well-defined grammar that evolves
with the language, clear rules about what an attribute appertains to,
and so on.

This series therefore adds keyword “attributes” that can appear
exactly where a standard attribute can appear and that appertain
to exactly what a standard attribute would appertain to.  The link is
mechanical and no opt-outs or variations are allowed.  This should
make the keywords predictable for programmers who are already
familiar with standard attributes.

This does mean that these keywords will be accepted for parsing purposes
in many more places than necessary.  Inappropriate uses will then be
diagnosed during semantic analysis.  However, the compiler would need
to reject the keywords in those positions whatever happens, and treating
them as ostensible attributes shouldn't be any worse than the alternative.
In some cases it might even be better.  For example, SME's
__arm_streaming attribute would make conceptual sense as a statement
attribute, so someone who takes a “try-it-and-see” approach might write:

  __arm_streaming { …block-of-code…; }

In fact, we did consider supporting this originally.  The reason for
rejecting it was that it was too difficult to implement, rather than
because it didn't make conceptual sense.

One slight disadvantage of the keyword-based approach is that it isn't
possible to use #pragma clang attribute with the keywords.  Perhaps we
could add support for that in future, if it turns out to be useful.

For want of a better term, I've called the new attributes "regular"
keyword attributes (in the sense that their parsing is regular wrt
standard attributes), as opposed to "custom" keyword attributes that
have their own parsing rules.

This patch adds the Attr.td support for regular keyword attributes.
Adding an attribute with a RegularKeyword spelling causes tablegen
to define the associated tokens and to record that attributes created
with that syntax are regular keyword attributes rather than custom
keyword attributes.

A follow-on patch contains the main Parse and Sema support,
which is enabled automatically by the Attr.td definition.

Other notes:

* The series does not allow regular keyword attributes to take
arguments, but this could be added in future.

* I wondered about trying to use tablegen for
TypePrinter::printAttributedAfter too, but decided against it.
RegularKeyword is really a spelling-level classification rather
than an attribute-level classification, and in general, an attribute
could have both GNU and RegularKeyword spellings.  In contrast,
printAttributedAfter is only given the attribute kind and the type
that results from applying the attribute.  AFAIK, it doesn't have
access to the original attribute spelling.  This means that some
attribute-specific or type-specific knowledge might be needed
to print the attribute in the best way.

* Generating the tokens automatically from Attr.td means that
pseudo's libgrammar does now depend on tablegen.

* The patch uses the SME __arm_streaming attribute as an example
for testing purposes.  The attribute does not do anything at this
stage.  Later SME-specific patches will add proper semantics for it,
and add other SME-related keyword attributes.

Differential Revision: https://reviews.llvm.org/D148700

Author: rsandifo-arm

clang-tools-extra/pseudo/lib/grammar/CMakeLists.txt

@@ -1,10 +1,5 @@
 set(LLVM_LINK_COMPONENTS Support)
 
-# This library is used by the clang-pseudo-gen tool which runs at build time.
-# Dependencies should be minimal to avoid long dep paths in the build graph.
-# It does use clangBasic headers (tok::TokenKind), but linking is not needed.
-# We have no transitive dependencies on tablegen files.
-list(REMOVE_ITEM LLVM_COMMON_DEPENDS clang-tablegen-targets)
 add_clang_library(clangPseudoGrammar
   Grammar.cpp
   GrammarBNF.cpp

clang/docs/InternalsManual.rst

@@ -2798,12 +2798,12 @@ and then the semantic handling of the attribute.
 Parsing of the attribute is determined by the various syntactic forms attributes
 can take, such as GNU, C++11, and Microsoft style attributes, as well as other
 information provided by the table definition of the attribute. Ultimately, the
-parsed representation of an attribute object is an ``ParsedAttr`` object.
+parsed representation of an attribute object is a ``ParsedAttr`` object.
 These parsed attributes chain together as a list of parsed attributes attached
 to a declarator or declaration specifier. The parsing of attributes is handled
-automatically by Clang, except for attributes spelled as keywords. When
-implementing a keyword attribute, the parsing of the keyword and creation of the
-``ParsedAttr`` object must be done manually.
+automatically by Clang, except for attributes spelled as so-called “custom”
+keywords. When implementing a custom keyword attribute, the parsing of the
+keyword and creation of the ``ParsedAttr`` object must be done manually.
 
 Eventually, ``Sema::ProcessDeclAttributeList()`` is called with a ``Decl`` and
 a ``ParsedAttr``, at which point the parsed attribute can be transformed
@@ -2856,33 +2856,60 @@ may have a keyword spelling, as well as a C++11 spelling and a GNU spelling. An
 empty spelling list is also permissible and may be useful for attributes which
 are created implicitly. The following spellings are accepted:
 
-  ============  ================================================================
-  Spelling      Description
-  ============  ================================================================
-  ``GNU``       Spelled with a GNU-style ``__attribute__((attr))`` syntax and
-                placement.
-  ``CXX11``     Spelled with a C++-style ``[[attr]]`` syntax with an optional
-                vendor-specific namespace.
-  ``C2x``       Spelled with a C-style ``[[attr]]`` syntax with an optional
-                vendor-specific namespace.
-  ``Declspec``  Spelled with a Microsoft-style ``__declspec(attr)`` syntax.
-  ``Keyword``   The attribute is spelled as a keyword, and required custom
-                parsing.
-  ``GCC``       Specifies two or three spellings: the first is a GNU-style
-                spelling, the second is a C++-style spelling with the ``gnu``
-                namespace, and the third is an optional C-style spelling with
-                the ``gnu`` namespace. Attributes should only specify this
-                spelling for attributes supported by GCC.
-  ``Clang``     Specifies two or three spellings: the first is a GNU-style
-                spelling, the second is a C++-style spelling with the ``clang``
-                namespace, and the third is an optional C-style spelling with
-                the ``clang`` namespace. By default, a C-style spelling is
-                provided.
-  ``Pragma``    The attribute is spelled as a ``#pragma``, and requires custom
-                processing within the preprocessor. If the attribute is meant to
-                be used by Clang, it should set the namespace to ``"clang"``.
-                Note that this spelling is not used for declaration attributes.
-  ============  ================================================================
+  ==================  =========================================================
+  Spelling            Description
+  ==================  =========================================================
+  ``GNU``             Spelled with a GNU-style ``__attribute__((attr))``
+                      syntax and placement.
+  ``CXX11``           Spelled with a C++-style ``[[attr]]`` syntax with an
+                      optional vendor-specific namespace.
+  ``C2x``             Spelled with a C-style ``[[attr]]`` syntax with an
+                      optional vendor-specific namespace.
+  ``Declspec``        Spelled with a Microsoft-style ``__declspec(attr)``
+                      syntax.
+  ``CustomKeyword``   The attribute is spelled as a keyword, and requires
+                      custom parsing.
+  ``RegularKeyword``  The attribute is spelled as a keyword. It can be
+                      used in exactly the places that the standard
+                      ``[[attr]]`` syntax can be used, and appertains to
+                      exactly the same thing that a standard attribute
+                      would appertain to. Lexing and parsing of the keyword
+                      are handled automatically.
+  ``GCC``             Specifies two or three spellings: the first is a
+                      GNU-style spelling, the second is a C++-style spelling
+                      with the ``gnu`` namespace, and the third is an optional
+                      C-style spelling with the ``gnu`` namespace. Attributes
+                      should only specify this spelling for attributes
+                      supported by GCC.
+  ``Clang``           Specifies two or three spellings: the first is a
+                      GNU-style spelling, the second is a C++-style spelling
+                      with the ``clang`` namespace, and the third is an
+                      optional C-style spelling with the ``clang`` namespace.
+                      By default, a C-style spelling is provided.
+  ``Pragma``          The attribute is spelled as a ``#pragma``, and requires
+                      custom processing within the preprocessor. If the
+                      attribute is meant to be used by Clang, it should
+                      set the namespace to ``"clang"``. Note that this
+                      spelling is not used for declaration attributes.
+  ==================  =========================================================
+
+The C++ standard specifies that “any [non-standard attribute] that is not
+recognized by the implementation is ignored” (``[dcl.attr.grammar]``).
+The rule for C is similar. This makes ``CXX11`` and ``C2x`` spellings
+unsuitable for attributes that affect the type system, that change the
+binary interface of the code, or that have other similar semantic meaning.
+
+``RegularKeyword`` provides an alternative way of spelling such attributes.
+It reuses the production rules for standard attributes, but it applies them
+to plain keywords rather than to ``[[…]]`` sequences. Compilers that don't
+recognize the keyword are likely to report an error of some kind.
+
+For example, the ``ArmStreaming`` function type attribute affects
+both the type system and the binary interface of the function.
+It cannot therefore be spelled ``[[arm::streaming]]``, since compilers
+that don't understand ``arm::streaming`` would ignore it and miscompile
+the code. ``ArmStreaming`` is instead spelled ``__arm_streaming``, but it
+can appear wherever a hypothetical ``[[arm::streaming]]`` could appear.
 
 Subjects
 ~~~~~~~~

clang/include/clang/Basic/Attr.td

@@ -315,6 +315,13 @@ class Keyword<string name, bit hasOwnParseRules>
     : Spelling<name, "Keyword"> {
   bit HasOwnParseRules = hasOwnParseRules;
 }
+
+// A keyword that can appear wherever a standard attribute can appear,
+// and that appertains to whatever a standard attribute would appertain to.
+// This is useful for things that affect semantics but that should otherwise
+// be treated like standard attributes.
+class RegularKeyword<string name> : Keyword<name, 0> {}
+
 // A keyword that has its own individual parsing rules.
 class CustomKeyword<string name> : Keyword<name, 1> {}
 
@@ -2427,6 +2434,11 @@ def AArch64SVEPcs: DeclOrTypeAttr {
   let Documentation = [AArch64SVEPcsDocs];
 }
 
+def ArmStreaming : TypeAttr, TargetSpecificAttr<TargetAArch64> {
+  let Spellings = [RegularKeyword<"__arm_streaming">];
+  let Documentation = [ArmStreamingDocs];
+}
+
 def Pure : InheritableAttr {
   let Spellings = [GCC<"pure">];
   let Documentation = [Undocumented];

clang/include/clang/Basic/AttrDocs.td

@@ -6551,6 +6551,41 @@ Requirements on Development Tools - Engineering Specification Documentation
   }];
 }
 
+def ArmStreamingDocs : Documentation {
+  let Category = DocCatType;
+  let Content = [{
+.. Note:: This attribute has not been implemented yet, but once it is
+   implemented, it will behave as described below.
+
+The ``__arm_streaming`` keyword is only available on AArch64 targets.
+It applies to function types and specifies that the function has a
+“streaming interface”.  This means that:
+
+* the function requires the Scalable Matrix Extension (SME)
+
+* the function must be entered in streaming mode (that is, with PSTATE.SM
+  set to 1)
+
+* the function must return in streaming mode
+
+See `Procedure Call Standard for the Arm® 64-bit Architecture (AArch64)
+<https://github.com/ARM-software/abi-aa>`_ for more details about
+streaming-interface functions.
+
+Clang manages PSTATE.SM automatically; it is not the source code's
+responsibility to do this.  For example, if a normal non-streaming
+function calls an ``__arm_streaming`` function, Clang generates code
+that switches into streaming mode before calling the function and
+switches back to non-streaming mode on return.
+
+``__arm_streaming`` can appear anywhere that a standard ``[[…]]`` type
+attribute can appear.
+
+See `Arm C Language Extensions <https://github.com/ARM-software/acle>`_
+for more details about this extension, and for other related SME features.
+  }];
+}
+
 def AlwaysInlineDocs : Documentation {
   let Category = DocCatFunction;
   let Content = [{

clang/include/clang/Basic/AttributeCommonInfo.h

@@ -77,6 +77,7 @@ class AttributeCommonInfo {
   unsigned SyntaxUsed : 4;
   unsigned SpellingIndex : 4;
   unsigned IsAlignas : 1;
+  unsigned IsRegularKeywordAttribute : 1;
 
 protected:
   static constexpr unsigned SpellingNotCalculated = 0xf;
@@ -86,24 +87,29 @@ class AttributeCommonInfo {
   /// including its syntax and spelling.
   class Form {
   public:
-    constexpr Form(Syntax SyntaxUsed, unsigned SpellingIndex, bool IsAlignas)
+    constexpr Form(Syntax SyntaxUsed, unsigned SpellingIndex, bool IsAlignas,
+                   bool IsRegularKeywordAttribute)
         : SyntaxUsed(SyntaxUsed), SpellingIndex(SpellingIndex),
-          IsAlignas(IsAlignas) {}
+          IsAlignas(IsAlignas),
+          IsRegularKeywordAttribute(IsRegularKeywordAttribute) {}
     constexpr Form(tok::TokenKind Tok)
         : SyntaxUsed(AS_Keyword), SpellingIndex(SpellingNotCalculated),
-          IsAlignas(Tok == tok::kw_alignas) {}
+          IsAlignas(Tok == tok::kw_alignas),
+          IsRegularKeywordAttribute(tok::isRegularKeywordAttribute(Tok)) {}
 
     Syntax getSyntax() const { return Syntax(SyntaxUsed); }
     unsigned getSpellingIndex() const { return SpellingIndex; }
     bool isAlignas() const { return IsAlignas; }
+    bool isRegularKeywordAttribute() const { return IsRegularKeywordAttribute; }
 
     static Form GNU() { return AS_GNU; }
     static Form CXX11() { return AS_CXX11; }
     static Form C2x() { return AS_C2x; }
     static Form Declspec() { return AS_Declspec; }
     static Form Microsoft() { return AS_Microsoft; }
-    static Form Keyword(bool IsAlignas) {
-      return Form(AS_Keyword, SpellingNotCalculated, IsAlignas);
+    static Form Keyword(bool IsAlignas, bool IsRegularKeywordAttribute) {
+      return Form(AS_Keyword, SpellingNotCalculated, IsAlignas,
+                  IsRegularKeywordAttribute);
     }
     static Form Pragma() { return AS_Pragma; }
     static Form ContextSensitiveKeyword() { return AS_ContextSensitiveKeyword; }
@@ -113,11 +119,12 @@ class AttributeCommonInfo {
   private:
     constexpr Form(Syntax SyntaxUsed)
         : SyntaxUsed(SyntaxUsed), SpellingIndex(SpellingNotCalculated),
-          IsAlignas(0) {}
+          IsAlignas(0), IsRegularKeywordAttribute(0) {}
 
     unsigned SyntaxUsed : 4;
     unsigned SpellingIndex : 4;
     unsigned IsAlignas : 1;
+    unsigned IsRegularKeywordAttribute : 1;
   };
 
   AttributeCommonInfo(const IdentifierInfo *AttrName,
@@ -127,7 +134,8 @@ class AttributeCommonInfo {
         ScopeLoc(ScopeLoc), AttrKind(AttrKind),
         SyntaxUsed(FormUsed.getSyntax()),
         SpellingIndex(FormUsed.getSpellingIndex()),
-        IsAlignas(FormUsed.isAlignas()) {
+        IsAlignas(FormUsed.isAlignas()),
+        IsRegularKeywordAttribute(FormUsed.isRegularKeywordAttribute()) {
     assert(SyntaxUsed >= AS_GNU && SyntaxUsed <= AS_Implicit &&
            "Invalid syntax!");
   }
@@ -154,7 +162,10 @@ class AttributeCommonInfo {
 
   Kind getParsedKind() const { return Kind(AttrKind); }
   Syntax getSyntax() const { return Syntax(SyntaxUsed); }
-  Form getForm() const { return Form(getSyntax(), SpellingIndex, IsAlignas); }
+  Form getForm() const {
+    return Form(getSyntax(), SpellingIndex, IsAlignas,
+                IsRegularKeywordAttribute);
+  }
   const IdentifierInfo *getAttrName() const { return AttrName; }
   SourceLocation getLoc() const { return AttrRange.getBegin(); }
   SourceRange getRange() const { return AttrRange; }
@@ -191,6 +202,8 @@ class AttributeCommonInfo {
     return SyntaxUsed == AS_Keyword || SyntaxUsed == AS_ContextSensitiveKeyword;
   }
 
+  bool isRegularKeywordAttribute() const { return IsRegularKeywordAttribute; }
+
   bool isContextSensitiveKeywordAttribute() const {
     return SyntaxUsed == AS_ContextSensitiveKeyword;
   }

clang/include/clang/Basic/CMakeLists.txt

@@ -35,6 +35,12 @@ clang_tablegen(AttrSubMatchRulesList.inc -gen-clang-attr-subject-match-rule-list
   SOURCE Attr.td
   TARGET ClangAttrSubjectMatchRuleList)
 
+clang_tablegen(AttrTokenKinds.inc -gen-clang-attr-token-kinds
+  -I ${CMAKE_CURRENT_SOURCE_DIR}/../../
+  SOURCE Attr.td
+  TARGET ClangAttrTokenKinds
+  )
+
 clang_tablegen(AttrHasAttributeImpl.inc -gen-clang-attr-has-attribute-impl
   -I ${CMAKE_CURRENT_SOURCE_DIR}/../../
   SOURCE Attr.td

clang/include/clang/Basic/TokenKinds.def

@@ -748,6 +748,12 @@ KEYWORD(__builtin_bit_cast               , KEYALL)
 KEYWORD(__builtin_available              , KEYALL)
 KEYWORD(__builtin_sycl_unique_stable_name, KEYSYCL)
 
+// Keywords defined by Attr.td.
+#ifndef KEYWORD_ATTRIBUTE
+#define KEYWORD_ATTRIBUTE(X) KEYWORD(X, KEYALL)
+#endif
+#include "clang/Basic/AttrTokenKinds.inc"
+
 // Clang-specific keywords enabled only in testing.
 TESTING_KEYWORD(__unknown_anytype , KEYALL)
 

clang/include/clang/Basic/TokenKinds.h

@@ -99,6 +99,13 @@ bool isAnnotation(TokenKind K);
 /// Return true if this is an annotation token representing a pragma.
 bool isPragmaAnnotation(TokenKind K);
 
+inline constexpr bool isRegularKeywordAttribute(TokenKind K) {
+  return (false
+#define KEYWORD_ATTRIBUTE(X) || (K == tok::kw_##X)
+#include "clang/Basic/AttrTokenKinds.inc"
+  );
+}
+
 } // end namespace tok
 } // end namespace clang
 

clang/include/clang/Lex/Token.h

@@ -117,8 +117,13 @@ class Token {
   }
 
   /// Return true if this is any of tok::annot_* kind tokens.
-  bool isAnnotation() const {
-    return tok::isAnnotation(getKind());
+  bool isAnnotation() const { return tok::isAnnotation(getKind()); }
+
+  /// Return true if the token is a keyword that is parsed in the same
+  /// position as a standard attribute, but that has semantic meaning
+  /// and so cannot be a true attribute.
+  bool isRegularKeywordAttribute() const {
+    return tok::isRegularKeywordAttribute(getKind());
   }
 
   /// Return a source location identifier for the specified

clang/lib/AST/TypePrinter.cpp

@@ -1757,6 +1757,11 @@ void TypePrinter::printAttributedAfter(const AttributedType *T,
     return;
   }
 
+  if (T->getAttrKind() == attr::ArmStreaming) {
+    OS << "__arm_streaming";
+    return;
+  }
+
   OS << " __attribute__((";
   switch (T->getAttrKind()) {
 #define TYPE_ATTR(NAME)
@@ -1797,6 +1802,7 @@ void TypePrinter::printAttributedAfter(const AttributedType *T,
   case attr::CmseNSCall:
   case attr::AnnotateType:
   case attr::WebAssemblyFuncref:
+  case attr::ArmStreaming:
     llvm_unreachable("This attribute should have been handled already");
 
   case attr::NSReturnsRetained:

clang/lib/Sema/SemaDeclAttr.cpp

@@ -5328,6 +5328,9 @@ bool Sema::CheckCallingConvAttr(const ParsedAttr &Attrs, CallingConv &CC,
   case ParsedAttr::AT_AArch64SVEPcs:
     CC = CC_AArch64SVEPCS;
     break;
+  case ParsedAttr::AT_ArmStreaming:
+    CC = CC_C; // FIXME: placeholder until real SME support is added.
+    break;
   case ParsedAttr::AT_AMDGPUKernelCall:
     CC = CC_AMDGPUKernelCall;
     break;

clang/lib/Sema/SemaType.cpp

@@ -126,6 +126,7 @@ static void diagnoseBadTypeAttribute(Sema &S, const ParsedAttr &attr,
   case ParsedAttr::AT_VectorCall:                                              \
   case ParsedAttr::AT_AArch64VectorPcs:                                        \
   case ParsedAttr::AT_AArch64SVEPcs:                                           \
+  case ParsedAttr::AT_ArmStreaming:                                            \
   case ParsedAttr::AT_AMDGPUKernelCall:                                        \
   case ParsedAttr::AT_MSABI:                                                   \
   case ParsedAttr::AT_SysVABI:                                                 \
@@ -4895,8 +4896,10 @@ static TypeSourceInfo *GetFullTypeForDeclarator(TypeProcessingState &state,
     // If we're supposed to infer nullability, do so now.
     if (inferNullability && !inferNullabilityInnerOnlyComplete) {
       ParsedAttr::Form form =
-          inferNullabilityCS ? ParsedAttr::Form::ContextSensitiveKeyword()
-                             : ParsedAttr::Form::Keyword(false /*IsAlignAs*/);
+          inferNullabilityCS
+              ? ParsedAttr::Form::ContextSensitiveKeyword()
+              : ParsedAttr::Form::Keyword(false /*IsAlignAs*/,
+                                          false /*IsRegularKeywordAttribute*/);
       ParsedAttr *nullabilityAttr = Pool.create(
           S.getNullabilityKeyword(*inferNullability), SourceRange(pointerLoc),
           nullptr, SourceLocation(), nullptr, 0, form);
@@ -7710,6 +7713,8 @@ static Attr *getCCTypeAttr(ASTContext &Ctx, ParsedAttr &Attr) {
     return createSimpleAttr<AArch64VectorPcsAttr>(Ctx, Attr);
   case ParsedAttr::AT_AArch64SVEPcs:
     return createSimpleAttr<AArch64SVEPcsAttr>(Ctx, Attr);
+  case ParsedAttr::AT_ArmStreaming:
+    return createSimpleAttr<ArmStreamingAttr>(Ctx, Attr);
   case ParsedAttr::AT_AMDGPUKernelCall:
     return createSimpleAttr<AMDGPUKernelCallAttr>(Ctx, Attr);
   case ParsedAttr::AT_Pcs: {