Skip to content

Instantiating a template with a template instantiation (eg Template1<Template2<Type>>) causes us to generate bad bindings #446

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fitzgen opened this issue Jan 26, 2017 · 28 comments
Assignees
Labels

Comments

@fitzgen
Copy link
Member

fitzgen commented Jan 26, 2017

Input:

template <typename> struct a;
namespace JS {
template <typename T> using b = a<T>;
template <typename T> class Rooted { b<T> c; };
}

Output with --enable-cxx-namespaces -- --std=c++11:

/* automatically generated by rust-bindgen */

pub mod root {
    #[allow(unused_imports)]
    use self::super::root;
    #[repr(C)]
    #[derive(Debug, Copy)]
    pub struct a {
        pub _address: u8,
    }
    impl Clone for a {
        fn clone(&self) -> Self { *self }
    }
    pub mod JS {
        #[allow(unused_imports)]
        use self::super::super::root;
        pub type b<T> = root::a;
        #[repr(C)]
        #[derive(Debug, Copy, Clone)]
        pub struct Rooted<T> {
            pub c: root::JS::b<T>,
        }
    }
}

Compilation errors:

error[E0392]: parameter `T` is never used
  --> js.rs:20:27
   |
20 |         pub struct Rooted<T> {
   |                           ^ unused type parameter
   |
   = help: consider removing `T` or using a marker such as `std::marker::PhantomData`

error: aborting due to previous error

I'm not 100% sure what the best thing to do in this case is, but whatever we do the bindings we emit should compile.

This seems really familiar... I wonder if we regressed something recently?

@emilio
Copy link
Contributor

emilio commented Jan 26, 2017

Nope, this isn't as much due to specialization but to us skipping template parameters with empty names.

@emilio
Copy link
Contributor

emilio commented Jan 26, 2017

We could fix that, I guess. In the end of the day those parameters can't be referenced, so we could define them with the name we want.

@fitzgen
Copy link
Member Author

fitzgen commented Jan 26, 2017

Hmm, so maybe creduce ended up finding a different bug then the one I was tracking down, because I'm hitting something related to this in SpiderMonkey DEBUG (but not non-DEBUG) builds.

@fitzgen
Copy link
Member Author

fitzgen commented Jan 26, 2017

Yes, generating names sounds good to me: _bindgen_T_1 etc.

@fitzgen fitzgen self-assigned this Jan 31, 2017
@fitzgen fitzgen added the bug label Jan 31, 2017
@fitzgen
Copy link
Member Author

fitzgen commented Jan 31, 2017

For posterity: I have a fix to generate names for anonymous named types, but this uncovers another extant bug which we are hitting in tests/headers/issue-358.hpp. This is going to be a bit more difficult to fix than the names was.

@fitzgen
Copy link
Member Author

fitzgen commented Jan 31, 2017

Specifically, here is the AST clang gives us:

(kind: Namespace, spelling: JS, type: Invalid
	(kind: ClassTemplate, spelling: PersistentRooted, type: Invalid
		(kind: TemplateTypeParameter, spelling: , type: Unexposed
		)
	)
)
(kind: ClassTemplate, spelling: a, type: Invalid
	(kind: TemplateTypeParameter, spelling: , type: Unexposed
	)
	(kind: FieldDecl, spelling: b, type: Pointer
		(kind: TypeRef, spelling: a<type-parameter-0-0>, type: Unexposed
		)
	)
)
(kind: Namespace, spelling: JS, type: Invalid
	(kind: ClassTemplate, spelling: PersistentRooted, type: Invalid
		(kind: TemplateTypeParameter, spelling: c, type: Unexposed
		)
		(kind: C++ base class specifier, spelling: a<PersistentRooted<c> >, type: Unexposed
			(kind: TemplateRef, spelling: a, type: Invalid
			)
			(kind: TemplateRef, spelling: PersistentRooted, type: Invalid
			)
			(kind: TypeRef, spelling: c, type: Unexposed
			)
		)
	)
)

Note that PersistentRooted's base specifier has an unexposed type and three(!) children. We need to somehow interpret those three children as the nested specialization a<PersistentRooted<c>>. Right now we fall on our face and interpret it as a<_anon_named_type_0> where _anon_named_type_0 is the name we generated for the anonymous named type within a, but it is somehow leaking out T.T

The result is that we generate this code:

#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct JS_PersistentRooted<c> {
    pub _base: a<__anon_named_type_0>,
    pub _phantom_0: ::std::marker::PhantomData<c>,
}
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct a<__anon_named_type_0> {
    pub b: *mut a<__anon_named_type_0>,
}

When it should really be something more like this:

#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct JS_PersistentRooted<c> {
    pub _base: a<JS_PersistentRooted<c>>,
    pub _phantom_0: ::std::marker::PhantomData<c>,
}
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct a<__anon_named_type_0> {
    pub b: *mut a<__anon_named_type_0>,
}

@fitzgen
Copy link
Member Author

fitzgen commented Jan 31, 2017

Here is a simpler test case that fails right now, without my patches applied, and doesn't involve either of anonymous named types or inheritance:

template <typename T>
class List {
    List<T> *next;
};

template <typename T>
class PersistentRooted {
    List<PersistentRooted<T>> list;
};

@fitzgen
Copy link
Member Author

fitzgen commented Feb 1, 2017

I think specializing with any specialization is broken.

fitzgen added a commit to fitzgen/rust-bindgen that referenced this issue Feb 1, 2017
@fitzgen
Copy link
Member Author

fitzgen commented Feb 1, 2017

Here is a simpler test case that fails right now, without my patches applied, and doesn't involve either of anonymous named types or inheritance:

Interesting, if I modify this test case to have List<PersistentRooted>, rather than List<PersistentRooted<T>>, then we get a simpler AST out of libclang and end up generating correct Rust bindings.

@fitzgen
Copy link
Member Author

fitzgen commented Feb 1, 2017

Both of these also work:

using Self = PersistentRooted<T>;
List<Self> list;

using Self = PersistentRooted;
List<Self> list;

@fitzgen
Copy link
Member Author

fitzgen commented Feb 2, 2017

Ok, here is a brain dump of what I have so far.

I had to modify build_template_wrapper to also consider CXCursor_TemplateRef kinds in additon to CXCursor_TypeRef. I suspect there are more latent bugs where we should be considering things we currently are not.

I think I might need to extend Cursor::is_template_like to return true for CXCursor_TemplateRef, although this by itself is not enough.

When parsing the PersistentRooted<T> inside List<Persistent<T>> via BindgenContext::build_template_wrapper for the List<...>:

  • We have the PersistentRooted class template on the currently_parsed_stack. This makes sense since the List<PersistentRooted<T>> is the type of one of its fields.

  • We end up in BindgenContext::builtin_or_resolved_ty, but I can't get anything useful out of the Type nor the location cursor:

FITZGEN: build_template_wrapper: visit: c = Cursor(PersistentRooted kind: TemplateRef, loc: ./issue-358.hpp:11:10, usr: None, cur_type: Type(, kind: Invalid, decl: {:?}, canon: {:?}), specialized: None)
FITZGEN: from_ty_or_ref_with_id: ItemId(8) Type(, kind: Invalid, decl: {:?}, canon: {:?}), Some(Cursor(PersistentRooted kind: TemplateRef, loc: ./issue-358.hpp:11:10, usr: None, cur_type: Type(, kind: Invalid, decl: {:?}, canon: {:?}), specialized: None)), Some(ItemId(1))
FITZGEN: from_ty_or_ref_with_id: have not collected typerefs
FITZGEN: from_ty_or_ref_with_id: ty = Type(, kind: Invalid, decl: {:?}, canon: {:?})
FITZGEN: from_ty_or_ref_with_id: location = Some(Cursor(PersistentRooted kind: TemplateRef, loc: ./issue-358.hpp:11:10, usr: None, cur_type: Type(, kind: Invalid, decl: {:?}, canon: {:?}), specialized: None))
FITZGEN: builtin_or_resolved_ty: Type(, kind: Invalid, decl: {:?}, canon: {:?}), Some(Cursor(PersistentRooted kind: TemplateRef, loc: ./issue-358.hpp:11:10, usr: None, cur_type: Type(, kind: Invalid, decl: {:?}, canon: {:?}), specialized: None)), Some(ItemId(1))
FITZGEN: currently_parsed_types = [
    (
        Cursor(PersistentRooted kind: ClassTemplate, loc: ./issue-358.hpp:9:7, usr: Some("c:@ST>1#T@PersistentRooted"), cur_type: Type(, kind: Invalid, decl: {:?}, canon: {:?}), specialized: None),
        ItemId(
            4
        )
    )
]
FITZGEN: builtin_or_resolved_ty: ty.declaration = Cursor( kind: NoDeclFound, loc: builtin definitions, usr: None, cur_type: Type(, kind: Invalid, decl: {:?}, canon: {:?}), specialized: None)
FITZGEN: builtin_or_resolved_ty: ty.declaration is not valid
FITZGEN: builtin_or_resolved_ty: have a location: Cursor(PersistentRooted kind: TemplateRef, loc: ./issue-358.hpp:11:10, usr: None, cur_type: Type(, kind: Invalid, decl: {:?}, canon: {:?}), specialized: None)
FITZGEN: builtin_or_resolved_ty: location is template-like
FITZGEN: builtin_or_resolved_ty: location is a template-ref, lets punt some and cross our fingers that we can grab the template being specialized
FITZGEN: builtin_or_resolved_ty: declaration = Cursor(PersistentRooted kind: TemplateRef, loc: ./issue-358.hpp:11:10, usr: None, cur_type: Type(, kind: Invalid, decl: {:?}, canon: {:?}), specialized: None)
FITZGEN: builtin_or_resolved_ty: canonical_declaration = Cursor(PersistentRooted kind: TemplateRef, loc: ./issue-358.hpp:11:10, usr: None, cur_type: Type(, kind: Invalid, decl: {:?}, canon: {:?}), specialized: None)
FITZGEN: builtin_or_resolved_ty: finish w/ build_builtin_ty
FITZGEN: from_ty_or_ref_with_id: nope, unresolved

My best hope at the moment is to check the currently_parsed_stack and see if we can find the same item somehow, either by comparing types or locations or usr or even spelling (I know, so terrible), and then reusing that.

@fitzgen
Copy link
Member Author

fitzgen commented Feb 2, 2017

@emilio do you have any ideas or feedback ^ ? Thanks!

@emilio
Copy link
Contributor

emilio commented Feb 2, 2017

The problem with checking currently_parsed_stack is that it's likely not enough? I need to digest this, I'll probably come back with more input :P

@emilio emilio changed the title template specialization causes us to generate bindings that won't compile partial template specialization causes us to generate bindings that won't compile Feb 2, 2017
@fitzgen
Copy link
Member Author

fitzgen commented Feb 2, 2017

This isn't partial template specialization, as I understand it. All the templates involved in my test cases are fully specialized, with no free template parameters.

@emilio
Copy link
Contributor

emilio commented Feb 2, 2017

Well, it's partial in the sense that PersistentRooted is incomplete when used as a template parameter in a base class, right?

@emilio
Copy link
Contributor

emilio commented Feb 2, 2017

I believe LLVM would give us proper type info if it was complete at that point.

@fitzgen
Copy link
Member Author

fitzgen commented Feb 2, 2017

Well, it's partial in the sense that PersistentRooted is incomplete when used as a template parameter in a base class, right?

Ah ok, we are each parsing "partial template specialization" differently :)

I have always parsed it as "(partial (template specialization))" and it seems you parse it as "((partial template) specialization)".

I do not know which parsing is correct, although I guess both come up in C++...

@fitzgen
Copy link
Member Author

fitzgen commented Feb 2, 2017

Anyways, I've been expanding our clang AST dumping, because I am going crazy trying to find what is or isn't available. PR to land this extended dumping soon, but here is what we get for the failing test case here:

(
 kind = ClassTemplate
 spelling = "List"
 location = ./issue-358.hpp:4:7
 is-definition? true
 is-declaration? true
 is-anonymous? false
 is-inlined-function? false
 template-kind = ClassDecl
 usr = "c:@ST>1#T@List"

 type.kind = Invalid

    (
     kind = TemplateTypeParameter
     spelling = "T"
     location = ./issue-358.hpp:3:20
     is-definition? true
     is-declaration? true
     is-anonymous? false
     is-inlined-function? false
     usr = "c:issue-358.hpp@43"

     type.kind = Unexposed
     type.spelling = "T"
     type.is-variadic? false

     type.canonical.kind = Unexposed
     type.canonical.spelling = "type-parameter-0-0"
     type.canonical.is-variadic? false
    )
    (
     kind = FieldDecl
     spelling = "next"
     location = ./issue-358.hpp:5:14
     is-definition? true
     is-declaration? true
     is-anonymous? false
     is-inlined-function? false
     usr = "c:@ST>1#T@List@FI@next"

     type.kind = Pointer
     type.spelling = "List<T> *"
     type.is-variadic? false

     type.canonical.kind = Pointer
     type.canonical.spelling = "List<T> *"
     type.canonical.is-variadic? false

     type.canonical.pointee.kind = Unexposed
     type.canonical.pointee.spelling = "List<T>"
     type.canonical.pointee.is-variadic? false

     type.pointee.kind = Unexposed
     type.pointee.spelling = "List<T>"
     type.pointee.is-variadic? false

     type.pointee.canonical.kind = Unexposed
     type.pointee.canonical.spelling = "List<T>"
     type.pointee.canonical.is-variadic? false

        (
         kind = TemplateRef
         spelling = "List"
         location = ./issue-358.hpp:5:5
         is-definition? false
         is-declaration? false
         is-anonymous? false
         is-inlined-function? false

         definition.kind = ClassTemplate
         definition.spelling = "List"
         definition.location = ./issue-358.hpp:4:7
         definition.is-definition? true
         definition.is-declaration? true
         definition.is-anonymous? false
         definition.is-inlined-function? false
         definition.template-kind = ClassDecl
         definition.usr = "c:@ST>1#T@List"

         referenced.kind = ClassTemplate
         referenced.spelling = "List"
         referenced.location = ./issue-358.hpp:4:7
         referenced.is-definition? true
         referenced.is-declaration? true
         referenced.is-anonymous? false
         referenced.is-inlined-function? false
         referenced.template-kind = ClassDecl
         referenced.usr = "c:@ST>1#T@List"

         type.kind = Invalid
        )
        (
         kind = TypeRef
         spelling = "T"
         location = ./issue-358.hpp:5:10
         is-definition? false
         is-declaration? false
         is-anonymous? false
         is-inlined-function? false

         definition.kind = TemplateTypeParameter
         definition.spelling = "T"
         definition.location = ./issue-358.hpp:3:20
         definition.is-definition? true
         definition.is-declaration? true
         definition.is-anonymous? false
         definition.is-inlined-function? false
         definition.usr = "c:issue-358.hpp@43"

         referenced.kind = TemplateTypeParameter
         referenced.spelling = "T"
         referenced.location = ./issue-358.hpp:3:20
         referenced.is-definition? true
         referenced.is-declaration? true
         referenced.is-anonymous? false
         referenced.is-inlined-function? false
         referenced.usr = "c:issue-358.hpp@43"

         type.kind = Unexposed
         type.spelling = "T"
         type.is-variadic? false

         type.canonical.kind = Unexposed
         type.canonical.spelling = "type-parameter-0-0"
         type.canonical.is-variadic? false
        )
    )
)
(
 kind = ClassTemplate
 spelling = "PersistentRooted"
 location = ./issue-358.hpp:9:7
 is-definition? true
 is-declaration? true
 is-anonymous? false
 is-inlined-function? false
 template-kind = ClassDecl
 usr = "c:@ST>1#T@PersistentRooted"

 type.kind = Invalid

    (
     kind = TemplateTypeParameter
     spelling = "T"
     location = ./issue-358.hpp:8:20
     is-definition? true
     is-declaration? true
     is-anonymous? false
     is-inlined-function? false
     usr = "c:issue-358.hpp@101"

     type.kind = Unexposed
     type.spelling = "T"
     type.is-variadic? false

     type.canonical.kind = Unexposed
     type.canonical.spelling = "type-parameter-0-0"
     type.canonical.is-variadic? false
    )
    (
     kind = FieldDecl
     spelling = "root_list"
     location = ./issue-358.hpp:11:31
     is-definition? true
     is-declaration? true
     is-anonymous? false
     is-inlined-function? false
     usr = "c:@ST>1#T@PersistentRooted@FI@root_list"

     type.kind = Unexposed
     type.spelling = "List<PersistentRooted<T> >"
     type.is-variadic? false

     type.canonical.kind = Unexposed
     type.canonical.spelling = "List<PersistentRooted<T> >"
     type.canonical.is-variadic? false

     type.declaration.kind = ClassTemplate
     type.declaration.spelling = "List"
     type.declaration.location = ./issue-358.hpp:4:7
     type.declaration.is-definition? true
     type.declaration.is-declaration? true
     type.declaration.is-anonymous? false
     type.declaration.is-inlined-function? false
     type.declaration.template-kind = ClassDecl
     type.declaration.usr = "c:@ST>1#T@List"

        (
         kind = TemplateRef
         spelling = "List"
         location = ./issue-358.hpp:11:5
         is-definition? false
         is-declaration? false
         is-anonymous? false
         is-inlined-function? false

         definition.kind = ClassTemplate
         definition.spelling = "List"
         definition.location = ./issue-358.hpp:4:7
         definition.is-definition? true
         definition.is-declaration? true
         definition.is-anonymous? false
         definition.is-inlined-function? false
         definition.template-kind = ClassDecl
         definition.usr = "c:@ST>1#T@List"

         referenced.kind = ClassTemplate
         referenced.spelling = "List"
         referenced.location = ./issue-358.hpp:4:7
         referenced.is-definition? true
         referenced.is-declaration? true
         referenced.is-anonymous? false
         referenced.is-inlined-function? false
         referenced.template-kind = ClassDecl
         referenced.usr = "c:@ST>1#T@List"

         type.kind = Invalid
        )
        (
         kind = TemplateRef
         spelling = "PersistentRooted"
         location = ./issue-358.hpp:11:10
         is-definition? false
         is-declaration? false
         is-anonymous? false
         is-inlined-function? false

         definition.kind = ClassTemplate
         definition.spelling = "PersistentRooted"
         definition.location = ./issue-358.hpp:9:7
         definition.is-definition? true
         definition.is-declaration? true
         definition.is-anonymous? false
         definition.is-inlined-function? false
         definition.template-kind = ClassDecl
         definition.usr = "c:@ST>1#T@PersistentRooted"

         referenced.kind = ClassTemplate
         referenced.spelling = "PersistentRooted"
         referenced.location = ./issue-358.hpp:9:7
         referenced.is-definition? true
         referenced.is-declaration? true
         referenced.is-anonymous? false
         referenced.is-inlined-function? false
         referenced.template-kind = ClassDecl
         referenced.usr = "c:@ST>1#T@PersistentRooted"

         type.kind = Invalid
        )
        (
         kind = TypeRef
         spelling = "T"
         location = ./issue-358.hpp:11:27
         is-definition? false
         is-declaration? false
         is-anonymous? false
         is-inlined-function? false

         definition.kind = TemplateTypeParameter
         definition.spelling = "T"
         definition.location = ./issue-358.hpp:8:20
         definition.is-definition? true
         definition.is-declaration? true
         definition.is-anonymous? false
         definition.is-inlined-function? false
         definition.usr = "c:issue-358.hpp@101"

         referenced.kind = TemplateTypeParameter
         referenced.spelling = "T"
         referenced.location = ./issue-358.hpp:8:20
         referenced.is-definition? true
         referenced.is-declaration? true
         referenced.is-anonymous? false
         referenced.is-inlined-function? false
         referenced.usr = "c:issue-358.hpp@101"

         type.kind = Unexposed
         type.spelling = "T"
         type.is-variadic? false

         type.canonical.kind = Unexposed
         type.canonical.spelling = "type-parameter-0-0"
         type.canonical.is-variadic? false
        )
    )
)

@fitzgen
Copy link
Member Author

fitzgen commented Feb 2, 2017

(FWIW, https://en.wikipedia.org/wiki/Partial_template_specialization describes "(partial (template specialization))")

@fitzgen fitzgen changed the title partial template specialization causes us to generate bindings that won't compile Instantiating a template with a template instantiation (eg Template1<Template2<Type>>) causes us to generate bad bindings Feb 2, 2017
@fitzgen
Copy link
Member Author

fitzgen commented Feb 2, 2017

Really frustrating that libclang is returning -1 from clang_Cursor_getNumTemplateArguments() for all of the template instantiations!

@emilio
Copy link
Contributor

emilio commented Feb 2, 2017

Worth noting that this only happens when there are incomplete types, i.e., the following works fine:

template <typename T>
struct Wrapper {
  T foo;
};

template <typename U>
struct Another {
  U bar;
};

typedef Wrapper<Another<int>> foobar;

@emilio
Copy link
Contributor

emilio commented Feb 2, 2017

We can probably try to coerce libclang in returning the proper types though, I think that's going to be the easiest fix.

@fitzgen
Copy link
Member Author

fitzgen commented Feb 2, 2017

From libclang:

int clang_Cursor_getNumTemplateArguments(CXCursor C) {
  if (clang_getCursorKind(C) != CXCursor_FunctionDecl) {
    return -1;
  }

  ...
}

I'm not super familiar with their code base, but that just seems plain wrong. What about templated class declarations (CXCursor_ClassDecl)? Templated type aliases? Both have different CXCursorKinds than CXCursor_FunctionDecl...

@fitzgen
Copy link
Member Author

fitzgen commented Feb 2, 2017

Observation: if we are instantiating an incomplete template, then that template must be on our currently parsing stack, or else the input would be malformed C++ and clang would give a hard error before we can even attempt to create our ir.

@fitzgen
Copy link
Member Author

fitzgen commented Feb 3, 2017

Immediate plan of attack:

  • Check the stack of things we are currently parsing as a last ditch effort when we don't know what the heck we're looking at in build_template_wrapper.

  • Because we get template arguemnts in a flat list, rather than properly nested, we need to reconstruct the nesting. I'll attempt to do this via a reverse fold of the template arguments.

Longer term plan of attack:

  • Send patches upstream to clang to make all the libclang APIs dealing with templates a little bit more sane.

@fitzgen
Copy link
Member Author

fitzgen commented Feb 6, 2017

Ok! Getting a little bit close!

My local branch is now generating too many template instantiations instead of too few! :-P

/* automatically generated by rust-bindgen */

#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct List<T> {
    pub next: *mut List<List<T>>,
}
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct PersistentRooted<T> {
    pub root_list: List<List<PersistentRooted<T>>>,
}

@emilio
Copy link
Contributor

emilio commented Feb 6, 2017 via email

fitzgen added a commit to fitzgen/rust-bindgen that referenced this issue Feb 7, 2017
fitzgen added a commit to fitzgen/rust-bindgen that referenced this issue Feb 7, 2017
fitzgen added a commit to fitzgen/rust-bindgen that referenced this issue Feb 7, 2017
fitzgen added a commit to fitzgen/rust-bindgen that referenced this issue Feb 8, 2017
fitzgen added a commit to fitzgen/rust-bindgen that referenced this issue Feb 8, 2017
fitzgen added a commit to fitzgen/rust-bindgen that referenced this issue Feb 8, 2017
bors-servo pushed a commit that referenced this issue Feb 9, 2017
Template instantiation

I very much suspect this needs a rebase, but I haven't done it yet.

This is the majority of #446, although it doesn't have the name generation for anonymous named types patches yet (that will be a different PR).
@fitzgen
Copy link
Member Author

fitzgen commented Feb 10, 2017

Going to close this for fixing instantiations. Will open new issues for figuring out how to handle unnamed type parameters, and aliases that don't use their template parameters.

@fitzgen fitzgen closed this as completed Feb 10, 2017
fitzgen added a commit to fitzgen/rust-bindgen that referenced this issue Feb 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants