Avoid inference getting stuck when the expected type contains a union/intersection

When we type a method call, we infer constraints based on its expected
type before typing its arguments. This way, we can type these arguments
with a precise expected type. This works fine as long as the constraints
we infer based on the expected type are _necessary_ constraints, but in
general type inference can go further and infer _sufficient_
constraints, meaning that we might get stuck with a set of constraints
which does not allow the method arguments to be typed at all.

Since 8067b95 we work around the
problem by simply not propagating any constraint when the expected type
is a union, but this solution is incomplete:

- It only handles unions at the top-level, but the same problem can
  happen with unions in any covariant position (method b of or-inf.scala)
  as well as intersections in contravariant positions (and-inf.scala,
  i8378.scala)
- Even when a union appear at the top-level, there might be constraints
  we can propagate, for example if only one branch can possibly match
  (method c of or-inf.scala)

Thankfully, we already have a solution that works for all these
problems: `TypeComparer#either` is capable of inferring only necessary
constraints. So far, this was only done when inferring GADT bounds to
preserve soundness, this commit extends this to use the same logic
when constraining a method based on its expected type.

Fixes #8378 which I previously thought was unfixable :).

Author: smarter

compiler/src/dotty/tools/dotc/core/Mode.scala

@@ -60,6 +60,9 @@ object Mode {
    */
   val Printing: Mode = newMode(10, "Printing")
 
+  /** We are constraining a method based on its expected type. */
+  val ConstrainResult: Mode = newMode(11, "ConstrainResult")
+
   /** We are currently in a `viewExists` check. In that case, ambiguous
    *  implicits checks are disabled and we succeed with the first implicit
    *  found.

compiler/src/dotty/tools/dotc/core/TypeComparer.scala

@@ -1364,14 +1364,25 @@ class TypeComparer(initctx: Context) extends ConstraintHandling[AbsentContext] w
 
   /** Returns true iff the result of evaluating either `op1` or `op2` is true and approximates resulting constraints.
    *
-   *  If we're _not_ in GADTFlexible mode, we try to keep the smaller of the two constraints.
-   *  If we're _in_ GADTFlexible mode, we keep the smaller constraint if any, or no constraint at all.
+   *  If we're inferring GADT bounds or constraining a method based on its
+   *  expected type, we infer only the _necessary_ constraints, this means we
+   *  keep the smaller constraint if any, or no constraint at all. This is
+   *  necessary for GADT bounds inference to be sound. When constraining a
+   *  method, this avoid painting ourselves into a corner we cannot backtrack
+   *  out of, see or-inf.scala and and-inf.scala for examples.
    *
+   *  Otherwise, we infer _sufficient_ constraints: we try to keep the smaller of
+   *  the two constraints, but if never is smaller than the other, we just pick
+   *  the first one.
+   *
+   *  @see [[necessaryEither]] for the GADT / result type case
    *  @see [[sufficientEither]] for the normal case
-   *  @see [[necessaryEither]] for the GADTFlexible case
    */
   protected def either(op1: => Boolean, op2: => Boolean): Boolean =
-    if (ctx.mode.is(Mode.GadtConstraintInference)) necessaryEither(op1, op2) else sufficientEither(op1, op2)
+    if ctx.mode.is(Mode.GadtConstraintInference) || ctx.mode.is(Mode.ConstrainResult) then
+      necessaryEither(op1, op2)
+    else
+      sufficientEither(op1, op2)
 
   /** Returns true iff the result of evaluating either `op1` or `op2` is true,
    *  trying at the same time to keep the constraint as wide as possible.
@@ -1438,8 +1449,8 @@ class TypeComparer(initctx: Context) extends ConstraintHandling[AbsentContext] w
    *     T1 & T2 <:< T3
    *     T1 <:< T2 | T3
    *
-   *  Unlike [[sufficientEither]], this method is used in GADTFlexible mode, when we are attempting to infer GADT
-   *  constraints that necessarily follow from the subtyping relationship. For instance, if we have
+   *  Unlike [[sufficientEither]], this method is used in GADTConstraintInference mode, when we are attempting
+   *  to infer GADT constraints that necessarily follow from the subtyping relationship. For instance, if we have
    *
    *     enum Expr[T] {
    *       case IntExpr(i: Int) extends Expr[Int]
@@ -1466,6 +1477,9 @@ class TypeComparer(initctx: Context) extends ConstraintHandling[AbsentContext] w
    *
    *  then the necessary constraint is { A = Int }, but correctly inferring that is, as far as we know, too expensive.
    *
+   *  This method is also used in ConstrainResult mode to avoid inference getting stuck due to the lack of backtracking,
+   *  see or-inf.scala and and-inf.scala for examples.
+   *
    *  Method name comes from the notion that we are keeping the constraint which is necessary to satisfy both
    *  subtyping relationships.
    */

compiler/src/dotty/tools/dotc/typer/ProtoTypes.scala

@@ -59,17 +59,14 @@ object ProtoTypes {
       else ctx.test(testCompat)
     }
 
-    private def disregardProto(pt: Type)(implicit ctx: Context): Boolean = pt.dealias match {
-      case _: OrType => true
-        // Don't constrain results with union types, since comparison with a union
-        // type on the right might commit too early into one side.
-      case pt => pt.isRef(defn.UnitClass)
-    }
+    private def disregardProto(pt: Type)(implicit ctx: Context): Boolean =
+      pt.dealias.isRef(defn.UnitClass)
 
     /** Check that the result type of the current method
      *  fits the given expected result type.
      */
-    def constrainResult(mt: Type, pt: Type)(implicit ctx: Context): Boolean = {
+    def constrainResult(mt: Type, pt: Type)(implicit parentCtx: Context): Boolean = {
+      given ctx as Context = parentCtx.addMode(Mode.ConstrainResult)
       val savedConstraint = ctx.typerState.constraint
       val res = pt.widenExpr match {
         case pt: FunProto =>

tests/neg/union.scala

@@ -17,7 +17,7 @@ object O {
 
   val x: A = f(new A { }, new A)
 
-  val y1: A | B = f(new A { }, new B) // error
+  val y1: A | B = f(new A { }, new B) // ok
   val y2: A | B = f[A | B](new A { }, new B) // ok
 
   val z = if (???) new A{} else new B

tests/pos/and-inf.scala

@@ -0,0 +1,13 @@
+class A
+class B
+
+class Inv[T]
+class Contra[-T]
+
+class Test {
+  def foo[T, S](x: T, y: S): Contra[Inv[T] & Inv[S]] = ???
+  val a: A = new A
+  val b: B = new B
+
+  val x: Contra[Inv[A] & Inv[B]] = foo(a, b)
+}

tests/neg/i6565.scala renamed to tests/pos/i6565.scala

@@ -9,9 +9,9 @@ def (o: Lifted[O]) flatMap [O, U] (f: O => Lifted[U]): Lifted[U] = ???
 val error: Err = Err()
 
 lazy val ok: Lifted[String] = { // ok despite map returning a union
-  point("a").map(_ => if true then "foo" else error) // error
+  point("a").map(_ => if true then "foo" else error) // ok
 }
 
 lazy val bad: Lifted[String] = { // found Lifted[Object]
-  point("a").flatMap(_ => point("b").map(_ => if true then "foo" else error)) // error
-}
+  point("a").flatMap(_ => point("b").map(_ => if true then "foo" else error)) // ok
+}

tests/pos/i8378.scala

@@ -0,0 +1,17 @@
+trait Has[A]
+
+trait A
+trait B
+trait C
+
+trait ZLayer[-RIn, +E, +ROut]
+
+object ZLayer {
+  def fromServices[A0, A1, B](f: (A0, A1) => B): ZLayer[Has[A0] with Has[A1], Nothing, Has[B]] =
+    ???
+}
+
+val live: ZLayer[Has[A] & Has[B], Nothing, Has[C]] =
+  ZLayer.fromServices { (a: A, b: B) =>
+    new C {}
+  }

tests/pos/or-inf.scala

@@ -0,0 +1,14 @@
+object Test {
+
+  def a(lis: Set[Int] | Set[String]) = {}
+  a(Set(1))
+  a(Set(""))
+
+  def b(lis: List[Set[Int] | Set[String]]) = {}
+  b(List(Set(1)))
+  b(List(Set("")))
+
+  def c(x: Set[Any] | Array[Any]) = {}
+  c(Set(1))
+  c(Array(1))
+}