Auto merge of #100096 - compiler-errors:fn-return-must-be-sized, r=jackh726

a fn pointer doesn't implement `Fn`/`FnMut`/`FnOnce` if its return type isn't sized

I stumbled upon #83915 which hasn't received much attention recently, and I wanted to revive it since this is one existing soundness hole that seems pretty easy to fix.

I'm not actually sure that the [alternative approach described here](#83915 (comment)) is sufficient, given the `src/test/ui/function-pointer/unsized-ret.rs` example I provided below. Rebasing the branch mentioned in that comment and testing that UI test, it seems that we actually end up only observing that `str: !Sized` during monomorphization, whereupon we ICE. Even if we were to fix that ICE, ideally we'd be raising an error that a fn pointer is being used badly during _typecheck_ instead of monomorphization, hence adapting the original approach in #83915.

I am happy to close this if people would prefer we rebase the original PR and land that -- I am partly opening to be annoying and get people thinking about this unsoundness again ❤️ 😸

cc: `@estebank` and `@nikomatsakis`
r? types

Here's a link to the thread: https://rust-lang.zulipchat.com/#narrow/stream/144729-t-types/topic/PR.20.2383915/near/235421351 for more context.

Author: bors

Diff for: compiler/rustc_trait_selection/src/traits/select/confirmation.rs

@@ -618,7 +618,36 @@ impl<'cx, 'tcx> SelectionContext<'cx, 'tcx> {
         )
         .map_bound(|(trait_ref, _)| trait_ref);
 
-        let nested = self.confirm_poly_trait_refs(obligation, trait_ref)?;
+        let mut nested = self.confirm_poly_trait_refs(obligation, trait_ref)?;
+
+        // Confirm the `type Output: Sized;` bound that is present on `FnOnce`
+        let cause = obligation.derived_cause(BuiltinDerivedObligation);
+        // The binder on the Fn obligation is "less" important than the one on
+        // the signature, as evidenced by how we treat it during projection.
+        // The safe thing to do here is to liberate it, though, which should
+        // have no worse effect than skipping the binder here.
+        let liberated_fn_ty = self.infcx.replace_bound_vars_with_placeholders(obligation.self_ty());
+        let output_ty = self
+            .infcx
+            .replace_bound_vars_with_placeholders(liberated_fn_ty.fn_sig(self.tcx()).output());
+        let output_ty = normalize_with_depth_to(
+            self,
+            obligation.param_env,
+            cause.clone(),
+            obligation.recursion_depth,
+            output_ty,
+            &mut nested,
+        );
+        let tr = ty::Binder::dummy(ty::TraitRef::new(
+            self.tcx().require_lang_item(LangItem::Sized, None),
+            self.tcx().mk_substs_trait(output_ty, &[]),
+        ));
+        nested.push(Obligation::new(
+            cause,
+            obligation.param_env,
+            tr.to_poly_trait_predicate().to_predicate(self.tcx()),
+        ));
+
         Ok(ImplSourceFnPointerData { fn_ty: self_ty, nested })
     }
 

Diff for: src/test/ui/function-pointer/sized-ret-with-binder.rs

@@ -0,0 +1,15 @@
+// check-pass
+
+#![feature(unboxed_closures)]
+
+fn is_fn<T: for<'a> Fn<(&'a (),)>>() {}
+fn is_fn2<T: for<'a, 'b> Fn<(&'a &'b (),)>>() {}
+
+struct Outlives<'a, 'b>(std::marker::PhantomData<&'a &'b ()>);
+
+fn main() {
+    is_fn::<for<'a> fn(&'a ()) -> &'a ()>();
+    is_fn::<for<'a> fn(&'a ()) -> &'a dyn std::fmt::Debug>();
+    is_fn2::<for<'a, 'b> fn(&'a &'b ()) -> Outlives<'a, 'b>>();
+    is_fn2::<for<'a, 'b> fn(&'a &'b ()) -> (&'a (), &'a ())>();
+}

Diff for: src/test/ui/function-pointer/unsized-ret.rs

@@ -0,0 +1,14 @@
+#![feature(fn_traits)]
+#![feature(unboxed_closures)]
+
+fn foo<F: Fn<T>, T>(f: Option<F>, t: T) {
+    let y = (f.unwrap()).call(t);
+}
+
+fn main() {
+    foo::<fn() -> str, _>(None, ());
+    //~^ ERROR the size for values of type `str` cannot be known at compilation time
+
+    foo::<for<'a> fn(&'a ()) -> (dyn std::fmt::Display + 'a), _>(None, (&(),));
+    //~^ ERROR the size for values of type `(dyn std::fmt::Display + 'a)` cannot be known at compilation time
+}

Diff for: src/test/ui/function-pointer/unsized-ret.stderr

@@ -0,0 +1,35 @@
+error[E0277]: the size for values of type `str` cannot be known at compilation time
+  --> $DIR/unsized-ret.rs:9:27
+   |
+LL |     foo::<fn() -> str, _>(None, ());
+   |     --------------------- ^^^^ doesn't have a size known at compile-time
+   |     |
+   |     required by a bound introduced by this call
+   |
+   = help: within `fn() -> str`, the trait `Sized` is not implemented for `str`
+   = note: required because it appears within the type `fn() -> str`
+note: required by a bound in `foo`
+  --> $DIR/unsized-ret.rs:4:11
+   |
+LL | fn foo<F: Fn<T>, T>(f: Option<F>, t: T) {
+   |           ^^^^^ required by this bound in `foo`
+
+error[E0277]: the size for values of type `(dyn std::fmt::Display + 'a)` cannot be known at compilation time
+  --> $DIR/unsized-ret.rs:12:66
+   |
+LL |     foo::<for<'a> fn(&'a ()) -> (dyn std::fmt::Display + 'a), _>(None, (&(),));
+   |     ------------------------------------------------------------ ^^^^ doesn't have a size known at compile-time
+   |     |
+   |     required by a bound introduced by this call
+   |
+   = help: within `for<'a> fn(&'a ()) -> (dyn std::fmt::Display + 'a)`, the trait `for<'a> Sized` is not implemented for `(dyn std::fmt::Display + 'a)`
+   = note: required because it appears within the type `for<'a> fn(&'a ()) -> (dyn std::fmt::Display + 'a)`
+note: required by a bound in `foo`
+  --> $DIR/unsized-ret.rs:4:11
+   |
+LL | fn foo<F: Fn<T>, T>(f: Option<F>, t: T) {
+   |           ^^^^^ required by this bound in `foo`
+
+error: aborting due to 2 previous errors
+
+For more information about this error, try `rustc --explain E0277`.

Diff for: src/test/ui/type-alias-impl-trait/issue-53398-cyclic-types.stderr

@@ -1,10 +1,11 @@
-error[E0275]: overflow evaluating the requirement `fn() -> Foo {foo}: Sized`
+error[E0275]: overflow evaluating the requirement `Foo: Sized`
   --> $DIR/issue-53398-cyclic-types.rs:5:13
    |
 LL | fn foo() -> Foo {
    |             ^^^
    |
    = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`issue_53398_cyclic_types`)
+   = note: required because it appears within the type `fn() -> Foo {foo}`
 
 error: aborting due to previous error