Fix implicit scope without scalatest

[anatoliykmetyuk]

No description provided.

[odersky]

@milessabin I am trying to clean up implicit scope, with a spec and code that are aligned. Right now, they are quite different, I am afraid. Shapeless 3 breaks with the new scope rules. The new spec is as follows:

  /** The implicit scope of a type `tp` is the smallest set S of object references (i.e. TermRefs
   *  with Module symbol) such that
   *
   *  - If `tp` is a class reference, S contains a reference to the companion object of the class,
   *    if it exists, as well as the implicit scopes of all of `tp`'s parent class references.
   *  - If `tp` is an opaque type alias `p.A` of type `tp'`, S contains a reference to an object `A` defined in the
   *    same scope as the opaque type, if it exists, as well as the implicit scope of `tp'`.
   *  - If `tp` is a reference `p.T` to a class or opaque type alias, S also contains all object references
   *    on the prefix path `p`. Under Scala-2 mode, package objects of package references on `p` also
   *    count towards the implicit scope.
   *  - If `tp` is a (non-opaque)  alias of `tp'`, S contains the implicit scope of `tp'`.
   *  - If `tp` is a singleton type, S contains the implicit scope of its underlying type.
   *  - If `tp` is some other type, its implicit scope is the union of the implicit scopes of
   *    its parts (parts defined as in the spec).
   */

Unfortunately shapeless3 breaks with that new definition of implicit scope. Can you take a look and either come up with a fix for shapeless or a suggestion how to tweak the spec to make it compile as is? Thanks!

[milessabin]

@odersky sure. Can you give me a quick sketch of what's changed?

[odersky]

@odersky sure. Can you give me a quick sketch of what's changed?

The previous code was hard to figure out, that's why I changed it. So it's difficult to say now what changed. One difference I know of is that in a type like a.b.T the new implicit scope is

• the companion object of T
• implicit scopes of all base class instances of T
• a and b if they are objects.

Previously, the implicit scope would have included companion objects of all supertypes of a and b as well. The published spec is even more restrictive than the new rules.

[smarter]

If tp is an opaque type alias p.A of type tp', S contains a reference to an object A defined in the same scope as the opaque type, if it exists, as well as the implicit scope of tp'.

I guess this also solves #6716 ?

[odersky]

Yes this also solves #6716. It's the reason I started the refactoring.

Regarding shapeless, the first error I see is:

53 |  inline def derived[A] given (gen: K0.ProductGeneric[A]): Monoid[A] = monoidGen
   |                                                                                ^
   |no implicit argument of type shapeless.K0.ProductInstances[shapeless.Monoid, A] was found for parameter inst of method monoidGen in object Monoid

My question is: Where is the implicit that produces a shapeless.K0.ProductInstances[shapeless.Monoid, A] located? Should that location be in the implicit scope of shapeless.K0.ProductInstances[shapeless.Monoid, A]?

[odersky]

I think I have it. The implicit is in shapeless.K0, which looks it's in the implicit scope since it is a prefix of ProductInstances`. But the latter is defined as

 type ProductInstances[F[_], T] = ErasedProductInstances[F[T]]

An alias does not count towards the implicit scope according to the new rules, and neither does its prefix.
Let's see whether we can change that.

[odersky]

In fact, it looks very tricky to make prefixes of aliases count towards implicit scope. There's the danger of introducing ambiguities. For instance in the following:

object A {
  type B
  given b as B = ...
}
export A._
export given A._

This wil produce

type B = A.B
given b as B = A.b

If paths towards aliases count, that would produce an ambiguity between b and A.b.

I tried to make ProductInstances in K0 opaque. This compiles fine in shapeless/main and would add
shapeless.K0 to the implicit scope of K0.ProductInstances. But the error does not go away which seems to indicate that the missing implicit is somewhere else. But where?

[odersky]

I tried to make ProductInstances in K0 opaque. This compiles fine in shapeless/main and would add
shapeless.K0 to the implicit scope of K0.ProductInstances.

... but unfortunately would still not find the necessary implicit because it fails to recognize that the two types are aliases when implicits are searched.

I then tried to make paths of abstract types count and to make shapeless instances abstract types instead of alias types. I.e. like this:

  type Instances[F[_], T] >: ErasedInstances[F[T]] <: ErasedInstances[F[T]]
  type ProductInstances[F[_], T] >: ErasedProductInstances[F[T]] <:  ErasedProductInstances[F[T]]
  type CoproductInstances[F[_], T] >: ErasedCoproductInstances[F[T]] <: ErasedCoproductInstances[F[T]]

That causes the right implicit to be found. But unfortunately it causes errors afterwards in the inline expansion.

So, I don't know what to do. I don't think we want to include paths of alias types in the implicit scope, for two reasons:

• It may cause ambiguities (see my previous comments)
• It would be very fragile, since the typer might dealias in somewhat unpredictable ways which would cause candidates to be dropped from the implicit scope in unpredicatble ways.

@milessabin Can you think of another solution to make the necessary instances visible?

[milessabin]

@odersky sorry, my machine has been tied up running @OlivierBlanvillain's benchmarks ...

Yes, I suspected that might be the problem.

I think it might be possible to make those aliases opaque ... I'm not surprised that just doing that on it's own leads to follow-on errors, but they're probably fixable.

That said, now that we have companions for opaque aliases which can contribute to the implicit scope, I think the case for similar companions for transparent aliases is quite a bit more compelling, simply out of symmetry with the opaque case. If we did that, the fix would simply be to add companions for the aliases in K0 etc. and put the instances there.

If we don't go down that route, then I will in effect be simulating that with opaque aliases/companions and inline implicit conversions which undo the opacity.

[milessabin]

@odersky on your ambiguity example ... is it really an ambiguity?

On current master the following compiles just fine,

object A {
  type B = Int
  given b as B = 23
}

object Test {
  export A._
  export given A._

  implicitly[B]
}

What in this PR would render that ambiguous?

I do agree, however, with your point about fragility. I've seen situations where differences in the compiler's enthusiasm for dealiasing has resulted in differences in implicit resolution.

[milessabin]

@odersky as it happens, I would quite like to make the various Kn aliases opaque, so I'm happy to pursue that line.

The first obstacle seems to be the following: just making type K0.ProductInstances opaque and making the result type of K0.mkProductInstances be ProductInstances[F, T] results in a compile error of the form,

[error] -- [E007] Type Mismatch Error: /home/miles/projects/shapeless-ng/core/src/test/scala/shapeless/type-classes.scala:53:80 
[error] 53 |  inline def derived[A] given (gen: K0.ProductGeneric[A]): Monoid[A] = monoidGen
[error]    |                                                                                ^
[error]    |Found:    shapeless.K0.ProductInstances[shapeless.Monoid, shapeless.ISB]
[error]    |Required: shapeless.K0.ProductInstances[shapeless.Monoid, shapeless.ISB]

which suggests that we might have a double-vision problem of some sort. I'll see if I can minimize it.

[odersky]

@milessabin Yes, I just hit the same problem. If you turn -Yprint-debug on you'll see that one of the types is in reality a this-type:

   |Found:    K0.this.ProductInstances[[A] => shapeless.this.Monoid[A], shapeless.this.ISB]
   |Required: shapeless.this.K0.ProductInstances[[A] => shapeless.this.Monoid[A], 

The opaque alias is visible from K0.this but not from the external type shapeless.K0 and TypeComparer assumes that it can always follow aliases without losing solutions. So this needs to be fixed (as well as the diagnostics output problem).

[odersky]

So, I guess the type K0.this.ProductInstances should not show up here. We are outside K0, so that type is illegal. Normally, asSeenFrom would take care of it. I instrumented the code and found that the asSeenFroms called on ProductInstances do indeed the right thing. So the error seems to be that we are missing a call to an asSeenFrom somewhere...

[odersky]

I am stuck. There are too many moving parts here. It seems there is some combination of opaque types and inlining that makes an this type escape from the scope where it is legal. Miles, can you try to minimize to minimize the code so that we get a chance to go to the bottom of this?

[milessabin]

@odersky sure, I can do that :-)

[odersky]

I made some progress. Things are more complicated than it looked at first. We might have to outlaw combining opaque types and inline methods in the same scope. I'll open an issue shortly.

[milessabin]

@odersky it looks like your inline/opaque fix means we have a way to move forward. How do you want to sequence things? I'm happy to patch things up in shapeless following this PR, or alternatively you could put this change behind a flag and we could do it in two steps.

[odersky]

Superseded by #6832