Skip to content

Adapt type parameters of typed eta expansion according to expected variances #950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Nov 17, 2015

Conversation

odersky
Copy link
Contributor

@odersky odersky commented Nov 11, 2015

Fixes #916. Review by @retronym?

@DarkDimius
Copy link
Contributor

/rebuild

@odersky
Copy link
Contributor Author

odersky commented Nov 11, 2015

/rebuild

@DarkDimius
Copy link
Contributor

https://scala-ci.typesafe.com/job/dotty-master-validate-partest/670/consoleFull
https://scala-ci.typesafe.com/job/dotty-master-validate-partest/668/consoleFull
https://scala-ci.typesafe.com/job/dotty-master-validate-partest/667/consoleFull

the same test failed 3 times with same error. Does not look like random failure

[error] *** error while checking /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/OrderingConstraint.scala after phase restoreScopes ***
[error] class dotty.tools.dotc.reporting.Reporter$Error at /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/OrderingConstraint.scala:195: None of the alternatives of val <none> satisfies required predicate
[error] dotty.tools.dotc.reporting.Reporting$class.error(Reporter.scala:106)
[error] dotty.tools.dotc.core.Contexts$Context.error(Contexts.scala:53)
[error] dotty.tools.dotc.typer.ErrorReporting$.errorType(ErrorReporting.scala:22)
[error] dotty.tools.dotc.typer.ErrorReporting$.errorTree(ErrorReporting.scala:19)
[error] dotty.tools.dotc.typer.Typer$$anonfun$typed$2.apply(Typer.scala:1154)
[error] dotty.tools.dotc.typer.Typer$$anonfun$typed$2.apply(Typer.scala:1149)
[error] dotty.tools.dotc.reporting.Reporting$class.traceIndented(Reporter.scala:154)
[error] dotty.tools.dotc.core.Contexts$Context.traceIndented(Contexts.scala:53)
[error] dotty.tools.dotc.typer.Typer.typed(Typer.scala:1149)
[error] dotty.tools.dotc.typer.Typer.typedType(Typer.scala:1190)
[error] dotty.tools.dotc.typer.Typer$$anonfun$typedValDef$1.apply(Typer.scala:900)
[error] dotty.tools.dotc.typer.Typer$$anonfun$typedValDef$1.apply(Typer.scala:897)
[error] dotty.tools.dotc.util.Stats$.track(Stats.scala:36)
[error] dotty.tools.dotc.typer.Typer.typedValDef(Typer.scala:897)
[error] dotty.tools.dotc.typer.Typer.typedNamed$1(Typer.scala:1086)
[error] dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:1139)
[error] dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:92)
[error] dotty.tools.dotc.transform.TreeChecker$Checker.typedUnadapted(TreeChecker.scala:194)
[error] dotty.tools.dotc.typer.Typer$$anonfun$typed$2.apply(Typer.scala:1151)
[error] dotty.tools.dotc.typer.Typer$$anonfun$typed$2.apply(Typer.scala:1149)
[error] dotty.tools.dotc.reporting.Reporting$class.traceIndented(Reporter.scala:154)
[error] dotty.tools.dotc.core.Contexts$Context.traceIndented(Contexts.scala:53)
[error] dotty.tools.dotc.typer.Typer.typed(Typer.scala:1149)
[error] dotty.tools.dotc.typer.Typer$$anonfun$typedDefDef$1$$anonfun$21.apply(Typer.scala:912)
[error] dotty.tools.dotc.typer.Typer$$anonfun$typedDefDef$1$$anonfun$21.apply(Typer.scala:912)
[error] dotty.tools.dotc.core.Decorators$ListDecorator$.loop$1(Decorators.scala:51)
[error] dotty.tools.dotc.core.Decorators$ListDecorator$.mapconserve$extension(Decorators.scala:67)
[error] dotty.tools.dotc.core.Decorators$ListOfListDecorator$$anonfun$nestedMapconserve$extension$1.apply(Decorators.scala:125)
[error] dotty.tools.dotc.core.Decorators$ListOfListDecorator$$anonfun$nestedMapconserve$extension$1.apply(Decorators.scala:125)
[error] dotty.tools.dotc.core.Decorators$ListDecorator$.loop$1(Decorators.scala:51)
[error] <<<
[error] exception occurred while compiling /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Annotations.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Constants.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Constraint.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/ConstraintHandling.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/ConstraintRunInfo.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Contexts.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Decorators.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Definitions.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/DenotTransformers.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Denotations.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Flags.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Hashable.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/NameOps.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Names.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/OrderingConstraint.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Periods.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Phases.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Scopes.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Signature.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/StdNames.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Substituters.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/SymDenotations.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/SymbolLoaders.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Symbols.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/TypeApplications.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/TypeComparer.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/TypeErasure.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/TypeOps.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/TyperState.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Types.scala, /home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/core/Uniques.scala
[info] !! 187 - pos/core                                  [compilation failed]

case _ => NoSymbol
}
def adaptArg(arg: Type): Type = arg match {
case arg: TypeRef if arg.symbol.isLambdaTrait =>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to this change, but the definition of isLambdaTrait:

    final def isLambdaTrait(implicit ctx: Context): Boolean =
      isClass && name.startsWith(tpnme.LambdaPrefix)

Looks prone to falsely matching:

scala> class `Lambda|`
defined class Lambda$bar

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, we should restrict this to members of the scala package.

@odersky
Copy link
Contributor Author

odersky commented Nov 12, 2015

/rebuild

3 similar comments
@odersky
Copy link
Contributor Author

odersky commented Nov 12, 2015

/rebuild

@odersky
Copy link
Contributor Author

odersky commented Nov 12, 2015

/rebuild

@odersky
Copy link
Contributor Author

odersky commented Nov 12, 2015

/rebuild

@odersky
Copy link
Contributor Author

odersky commented Nov 12, 2015

Heisenbug hates me. Or at least is very good in hiding...

@retronym
Copy link
Member

@odersky Do you believe this bug is exposed by parallel test execution sharing a symbol table? Or is it a single-threaded non-determinism?

Assuming the former:

What sort of operations are supported concurrently?

I've reviewed a little of the caching code. This is prone to race condition:

    private def memberCache(implicit ctx: Context): LRUCache[Name, PreDenotation] = {
      if (myMemberCachePeriod != ctx.period) {
        myMemberCache = new LRUCache
        myMemberCachePeriod = ctx.period
      }
      myMemberCache
    }

Same story for LRUCache#enter:

  def enter(key: Key, value: Value): Unit = {
    keys(last) = key
    values(last) = value
    last = lastButOne
  }

@DarkDimius
Copy link
Contributor

@retronym, those methods should not be executed by different threads.
Partest gives a new ContextBase to every thread and this should mean that all symbols\types\trees\denotation are thread-private.
Though I've added a check to those places 7b94e98, just to be sure I'm not mistaking.

@odersky
Copy link
Contributor Author

odersky commented Nov 13, 2015

The latest additions to diagnostics have finally produced something
interesting. See partial log below. We are not finding a method in a scope,
even though the method (newRefArray) is present and the scope is
well-formed (as verified by checkConsistent). A "slow" (linear) search
finds the method but the initial hashtable lookup didn't. What's more,
several threads try to get the same method. So this seems to point to a
race condition. But as @DarkDimius says, scopes should not be shared
between threads. We'll hopefully find out soon whether that's indeed the
case.

One other suggestion: The failures we are seeing happen when running the
bootstrapped dotty compiler, right? Can we instead run the original one for
a while and see whether the errors go away? I would like to exclude that
our implementation of lazy vals has some cross-talk. The failures we are
seeing would be consistent with the hypothesis that the lazy vals in
Definitions somehow get shared between several Definition instances.

[info] *** missing method:
dotty.tools.dotc.core.Decorators$StringDecorator@1ddb9b46 in module
Arrays
[info] decls = Scope{
[info] def : ()dotty.runtime.Arrays$
[info] def newGenericArray: [T](length: Int)(implicit tag:
scala.reflect.ClassTag[T])Array[T]
[info] def seqToArray: [T](xs: scala.collection.Seq[T], clazz:
Class[])Array[T]
[info] def newRefArray: [T](length: Int)T
[info] def newByteArray: (length: Int)Array[Byte]
[info] def newShortArray: (length: Int)Array[Short]
[info] def newCharArray: (length: Int)Array[Char]
[info] def newIntArray: (length: Int)Array[Int]
[info] def newLongArray: (length: Int)Array[Long]
[info] def newFloatArray: (length: Int)Array[Float]
[info] def newDoubleArray: (length: Int)Array[Double]
[info] def newBooleanArray: (length: Int)Array[Boolean]
[info] def newUnitArray: (length: Int)Array[Unit]
[info] }
[info] newUnitArray
[info] newBooleanArray
[info] newDoubleArray
[info] newFloatArray
[info] newLongArray
[info] newIntArray
[info] newCharArray
[info] newShortArray
[info] newByteArray
[info] newRefArray
[info] seqToArray
[info] newGenericArray
[info]
[info] *** missing method:
dotty.tools.dotc.core.Decorators$StringDecorator@1ddb9b46 in module
Arrays
[info] *** missing method:
dotty.tools.dotc.core.Decorators$StringDecorator@1ddb9b46 in module
Arrays
[info] decls = Scope{
[info] def : ()dotty.runtime.Arrays$
[info] def newGenericArray: [T](length: Int)(implicit tag:
scala.reflect.ClassTag[T])Array[T]
[info] def seqToArray: [T](xs: scala.collection.Seq[T], clazz:
Class[
])Array[T]
[info] def newRefArray: [T](length: Int)T
[info] def newByteArray: (length: Int)Array[Byte]
[info] def newShortArray: (length: Int)Array[Short]
[info] def newCharArray: (length: Int)Array[Char]
[info] def newIntArray: (length: Int)Array[Int]
[info] def newLongArray: (length: Int)Array[Long]
[info] def newFloatArray: (length: Int)Array[Float]
[info] def newDoubleArray: (length: Int)Array[Double]
[info] def newBooleanArray: (length: Int)Array[Boolean]
[info] def newUnitArray: (length: Int)Array[Unit]
[info] }
[info] decls = Scope{
[info] def : ()dotty.runtime.Arrays$
[info] def newGenericArray: [T](length: Int)(implicit tag:
scala.reflect.ClassTag[T])Array[T]
[info] def seqToArray: [T](xs: scala.collection.Seq[T], clazz:
Class[_])Array[T]
[info] def newRefArray: [T](length: Int)T
[info] def newByteArray: (length: Int)Array[Byte]
[info] def newShortArray: (length: Int)Array[Short]
[info] def newCharArray: (length: Int)Array[Char]
[info] def newIntArray: (length: Int)Array[Int]
[info] def newLongArray: (length: Int)Array[Long]
[info] def newFloatArray: (length: Int)Array[Float]
[info] def newDoubleArray: (length: Int)Array[Double]
[info] def newBooleanArray: (length: Int)Array[Boolean]
[info] def newUnitArray: (length: Int)Array[Unit]
[info] }
[info] newUnitArray
[info] newBooleanArray
[info] newDoubleArray
[info] newUnitArray
[info] newBooleanArray
[info] newDoubleArray
[info] newFloatArray
[info] newLongArray
[info] newIntArray
[info] newCharArray
[info] newShortArray
[info] newByteArray
[info] newRefArray
[info] seqToArray
[info] newGenericArray
[info]
[info] newFloatArray
[info] newLongArray
[info] newIntArray
[info] newCharArray
[info] newShortArray
[info] newByteArray
[info] newRefArray
[info] seqToArray
[info] newGenericArray
[info]
[error] **** slow search found: method newRefArray
[error] scope entries found:
[error] no more entries found!
[error] **** slow search found: method newRefArray
[error] scope entries found:
[error] **** slow search found: method newRefArray
[error] scope entries found:
[error] no more entries found!
[error] no more entries found!
[error] exception occurred while compiling
/home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/reporting/ConsoleReporter.scala,
/home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/reporting/Reporter.scala,
/home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/reporting/StoreReporter.scala,
/home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/reporting/ThrowingReporter.scala,
/home/jenkins/workspace/dotty-master-validate-partest/tests/partest-generated/pos/reporting/UniqueMessagePositions.scala
[error] dotty.tools.dotc.core.Types$TypeError: Missing method: module
Arrays . dotty.tools.dotc.core.Decorators$StringDecorator@1ddb9b46
[error] at dotty.tools.dotc.core.Denotations$Denotation.requiredMethod(Denotations.scala:229)
[error] at dotty.tools.dotc.core.Definitions.newRefArrayMethod(Definitions.scala:245)
[error] at dotty.tools.dotc.core.Definitions.isPolymorphicAfterErasure(Definitions.scala:608)
[error] at dotty.tools.dotc.core.TypeErasure$.transformInfo(TypeErasure.scala:174)
[error] at dotty.tools.dotc.transform.Erasure.transform(Erasure.scala:54)
[error] at dotty.tools.dotc.core.Denotations$SingleDenotation.currentIfExists(Denotations.scala:665)

On Fri, Nov 13, 2015 at 8:32 AM, Dmitry Petrashko [email protected]
wrote:

@retronym https://github.com/retronym, those methods should be executed
by different threads.
Partest gives a new ContextBase to every thread and this should mean that
all symbols\types\trees\denotation are thread-private.
Though I've added a check to those places 7b94e98
7b94e98,
just to be sure I'm not mistaking.


Reply to this email directly or view it on GitHub
#950 (comment).

Martin Odersky
EPFL

@DarkDimius
Copy link
Contributor

@odersky Looking at implementation of MutableScope, I've found a huge inefficiency, though I do not see how it will lead to lookup failures: #953.

@retronym
Copy link
Member

What is shared? The name table?
On Fri, 13 Nov 2015 at 18:01, Dmitry Petrashko [email protected]
wrote:

@odersky https://github.com/odersky Looking at implementation of
MutableScope, I've found a huge inefficiency, though I do not see how it
will lead to lookup failures: #953
#953.


Reply to this email directly or view it on GitHub
#950 (comment).

@odersky
Copy link
Contributor Author

odersky commented Nov 13, 2015

Yes, nametable is the only data structure that is shared.

@odersky
Copy link
Contributor Author

odersky commented Nov 16, 2015

/rebuild

1 similar comment
@odersky
Copy link
Contributor Author

odersky commented Nov 16, 2015

/rebuild

When eta expanding a type `C` to `[vX] => C[X]` the variance `v`
is now the variance of the expected type, not the variance of the
type constructor `C`, as long as the variance of the expected type
is compatible with the variance of `C`.
Exclude false positives such as `Lambda|` be requiring
that lambda traits are defined in the Scala package.
@odersky
Copy link
Contributor Author

odersky commented Nov 17, 2015

rebased to master

@odersky
Copy link
Contributor Author

odersky commented Nov 17, 2015

Wow. Is the Heisenbug slayed, finally?

getSimpleName crashes on some module names created by scalac.

May help finding the partest issue. (reverted from commit c11646c)
Adding parents signals (via SymDenotation.fullyDefined) that
the class can now be frozen. So this should be done only after all
members are entered.
Fix problems arising when hierarchies of classes are under completion at the same time.
In this case it can happen that we see a subclass (e.g. Arrays.scala) which depends
on a superclass (e.g. GenTraversableLike.scala) that itself does not have its parents
defined yet. Previously, several things went wrong here

 - One of the base classes would be marked as frozen, even though it dod not have all
   members entered yet. This led to an error in finger printing.
 - The baseclasses and super class bits of the subclass would be computed before the parents
   of the middle class were known. The baseclasses would then be chached, leading to
   false results for isDerivedFrom.

We need to refine the logic for computing base classes, super class bits, and fingerprints
to account for that issue.
The previous commit made packages always fully completed.
This is wrong - since new members can be added to packages
at any time, packages are never fully completed.
Just noted that the previous scope might have been too small.
We compute the bucket index with the table size before going into the
synchronized. But that might mean we see a stale table size.
Let's see what this gives us.
@odersky
Copy link
Contributor Author

odersky commented Nov 17, 2015

Removed all the added Heisenbug scaffolding for requiredMethod and force pushed.

@odersky
Copy link
Contributor Author

odersky commented Nov 17, 2015

Two successes in a row! I am taking the liberty to merge so that others can profit.

odersky added a commit that referenced this pull request Nov 17, 2015
Adapt type parameters of typed eta expansion according to expected variances
@odersky odersky merged commit eefa611 into scala:master Nov 17, 2015
@allanrenucci allanrenucci deleted the fix-#916 branch December 14, 2017 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants