java.lang.NullPointerException on pattern match in dotty REPL #4051

prayagupa · 2018-02-28T19:26:48Z

Might be more of stackoverflow question not a issues at all but I was playing with dotty REPL with a tuple2 and _._2 works fine

scala> Seq(("Porcupine", 1), ("Floyd", 2), ("Phil Collins", 3)).map(_._2) 
val res0: Seq[Int] = List(1, 2, 3)

But when I pattern match and get the second column it throws a NullPointerException.

scala> Seq(("Porcupine", 1), ("Floyd", 2), ("Phil Collins", 3)).collect {case (name, value) => value } 
Exception in thread "main" java.lang.NullPointerException
	at dotty.tools.dotc.transform.ExpandPrivate.ensurePrivateAccessible(ExpandPrivate.scala:93)
	at dotty.tools.dotc.transform.ExpandPrivate.transformSelect(ExpandPrivate.scala:104)
	at dotty.tools.dotc.transform.MegaPhase.goSelect(MegaPhase.scala:531)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:212)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:364)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:248)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:365)
	at dotty.tools.dotc.transform.MegaPhase.mapDefDef$1(MegaPhase.scala:228)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:231)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:364)
	at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:373)
	at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:378)
	at scala.collection.immutable.List.mapConserve(List.scala:176)
	at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:378)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:317)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:365)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:234)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:364)
	at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:373)
	at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:378)
	at scala.collection.immutable.List.mapConserve(List.scala:176)
	at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:378)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:317)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:365)
	at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:234)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:364)
	at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:373)
	at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:378)
	at scala.collection.immutable.List.mapConserve(List.scala:176)
	at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:378)
	at dotty.tools.dotc.transform.MegaPhase.mapPackage$1(MegaPhase.scala:334)
	at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:337)
	at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:365)
	at dotty.tools.dotc.transform.MegaPhase.transformUnit(MegaPhase.scala:384)
	at dotty.tools.dotc.transform.MegaPhase.run(MegaPhase.scala:396)
	at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:293)
	at scala.collection.immutable.List.map(List.scala:283)
	at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:295)
	at dotty.tools.dotc.Run.runPhases$4$$anonfun$4(Run.scala:123)
	at scala.compat.java8.JProcedure1.apply(JProcedure1.java:18)
	at scala.compat.java8.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:191)
	at dotty.tools.dotc.Run.runPhases$5(Run.scala:136)
	at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:141)
	at scala.compat.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
	at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:88)
	at dotty.tools.dotc.Run.compileUnits(Run.scala:143)
	at dotty.tools.dotc.Run.compileUnits(Run.scala:99)
	at dotty.tools.repl.ReplCompiler.runCompilationUnit(ReplCompiler.scala:188)
	at dotty.tools.repl.ReplCompiler.compile(ReplCompiler.scala:197)
	at dotty.tools.repl.ReplDriver.compile(ReplDriver.scala:223)
	at dotty.tools.repl.ReplDriver.interpret(ReplDriver.scala:196)
	at dotty.tools.repl.ReplDriver.runUntilQuit(ReplDriver.scala:149)
	at dotty.tools.repl.Main$.main(Main.scala:6)
	at dotty.tools.repl.Main.main(Main.scala)

Get the same error with Seq(("Porcupine", 1), ("Floyd", 2), ("Phil Collins", 3)).collect {case x => x._2}.

Tried a bit exploring the reason which is in ExpandPrivate#ensurePrivateAccessible which is line assert(d.symbol.sourceFile != null && isSimilar(d.symbol.sourceFile.path, ctx.owner.sourceFile.path)

  private def ensurePrivateAccessible(d: SymDenotation)(implicit ctx: Context) =
    if (isVCPrivateParamAccessor(d))
      d.ensureNotPrivate.installAfter(thisPhase)
    else if (d.is(PrivateTerm) && !d.owner.is(Package) && d.owner != ctx.owner.lexicallyEnclosingClass) {
      // Paths `p1` and `p2` are similar if they have a common suffix that follows
      // possibly different directory paths. That is, their common suffix extends
      // in both cases either to the start of the path or to a file separator character.
      def isSimilar(p1: String, p2: String): Boolean = {
        var i = p1.length - 1
        var j = p2.length - 1
        while (i >= 0 && j >= 0 && p1(i) == p2(j) && p1(i) != separatorChar) {
          i -= 1
          j -= 1
        }
        (i < 0 || p1(i) == separatorChar) &&
        (j < 0 || p1(j) == separatorChar)
      }

      assert(d.symbol.sourceFile != null &&
             isSimilar(d.symbol.sourceFile.path, ctx.owner.sourceFile.path),
          s"private ${d.symbol.showLocated} in ${d.symbol.sourceFile} accessed from ${ctx.owner.showLocated} in ${ctx.owner.sourceFile}")
      d.ensureNotPrivate.installAfter(thisPhase)
    }

I also tried with simpler pattern match, it does not work.

Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 10).collect {case x if x % 2 == 0 => (x, 1) }

Don't know what is SymDenotation.symbol or Context.owner. I installed dotty using homebrew and believe it is the built from latest codebase.

The text was updated successfully, but these errors were encountered:

Blaisorblade · 2018-02-28T23:22:12Z

Thanks for the report. That’s a crash in the compiler, those are always bugs. Dotty from homebrew is likely the latest release. Todo: recheck on latest master.

allanrenucci · 2018-03-01T09:03:02Z

Can be reproduced with List(1, 2, 3).collect { case x => x }. ctx.owner.sourceFile is null in the REPL: https://github.com/lampepfl/dotty/blob/8bd162554d266a23896eb0833a264b57873d3140/compiler/src/dotty/tools/dotc/transform/ExpandPrivate.scala#L93

expz · 2018-03-03T23:47:26Z

There is no error with an explicit PartialFunction:

scala> val id = new PartialFunction[Int, Int] {
         def apply(x: Int) = x
         def isDefinedAt(x: Int) = true
       } 
val id: PartialFunction[Int, Int] = <function1>
scala> Seq(1, 2, 3).collect(id) 
val res1: Seq[Int] = List(1, 2, 3)

But even an identity function fails:

scala> def id(x: Int) = x 
def id(x: Int): Int
scala> Seq(1, 2, 3).collect(id) 
Exception in thread "main" java.lang.NullPointerException
	at dotty.tools.dotc.transform.ExpandPrivate.ensurePrivateAccessible(ExpandPrivate.scala:93)
        ...

Blaisorblade · 2018-03-04T00:02:41Z

@allanrenucci is the sourcefile supposed to be null? If so, can we relax the assertion? It seems some of its checks should become proper conditions rather than an assert?

allanrenucci · 2018-03-04T11:08:40Z

By quickly looking at it, I would say sourcefile should not be null. When running within the REPL, the source file is a dotty.tools.io.VirtualFile. I am not sure why null is returned instead

expz · 2018-03-17T23:28:42Z

Here is the reason null is returned. An anonymous class rs$line$2$$anon$1 is defined and then instantiated as an object of type PartialFunction. The sourceFile of the anonymous class is null. This happens, because:

sourceFile first tries to read the assocFile of the ClassSymbol and finds that it is null.
So it tries to create the source file from the myAnnotations of (the denotation of) the ClassSymbol. Unfortunately, all the information the annotation (a SourceFile(Apply(...))) contains is the filename as a string. So sourceFile tries to recreate an AbstractFile by calling AbstractFile.getFile. In the case of the normal compiler, this works and returns the PlainFile of the scala source file. But in the case of the REPL, it tries to find rs$line$2 in the filesystem and fails, returning null.

I see two options. A hack to fix this is to just create a VirtualFile if the PlainFile cannot be created:

    final def sourceFile(implicit ctx: Context): AbstractFile = {
      val file = associatedFile
      if (file != null && !file.path.endsWith("class")) file
      else {
        val topLevelCls = denot.topLevelClass(ctx.withPhaseNoLater(ctx.flattenPhase))
        topLevelCls.getAnnotation(defn.SourceFileAnnot) match {
          case Some(sourceAnnot) => sourceAnnot.argumentConstant(0) match {
            case Some(Constant(path: String)) => // Changes here:
              val plainFile = AbstractFile.getFile(path)
              if (plainFile != null) plainFile
              else new dotty.tools.io.VirtualFile(path)
            case none => null
          }
          case none => null
        }
      }
    }

This solves the error and produces no errors in dotty-compiler/test or testCompilation. But it looks ugly to me. It introduces a dependency on VirtualFile where it seems out of place. And perhaps this code in dotty.tools.dotc.interactive.SourceTree relies on returning null for a non-existant path in the SourceFile(...) annotation?

object SourceTree {
  def fromSymbol(sym: ClassSymbol, id: String = "")(implicit ctx: Context): Option[SourceTree] = {
    if (sym == defn.SourceFileAnnot || // FIXME: No SourceFile annotation on SourceFile itself
        sym.sourceFile == null) // FIXME: We cannot deal with external projects yet
      None
    ...

The second option would be to decide that the anonymous class which becomes a PartialFunction object should have assocFile != null. But I do not know whether that matches the intended meaning of assocFile.

allanrenucci · 2018-03-18T13:44:13Z

I don't think associatedFile should return null. Whether the source is a normal file or a virtual one shouldn't matter.

Fix #4051: Add regression test

Blaisorblade added the itype:crash label Feb 28, 2018

allanrenucci added the area:repl label Mar 1, 2018

allanrenucci self-assigned this Mar 4, 2018

allanrenucci added the prio:high label Mar 4, 2018

allanrenucci added a commit to dotty-staging/dotty that referenced this issue Jul 13, 2018

Fix scala#4051: Add regression test

2867faf

allanrenucci closed this as completed in 673a646 Jul 16, 2018

allanrenucci added a commit that referenced this issue Jul 16, 2018

Merge pull request #4791 from dotty-staging/fix-4051

292c51c

Fix #4051: Add regression test

allanrenucci mentioned this issue Oct 31, 2018

REPL crash when compiling lambda to anonymous class #5345

Closed

allanrenucci mentioned this issue Jan 17, 2019

REPL will crash when expression SAM converting to abstract class type #5733

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

java.lang.NullPointerException on pattern match in dotty REPL #4051

java.lang.NullPointerException on pattern match in dotty REPL #4051

prayagupa commented Feb 28, 2018

Blaisorblade commented Feb 28, 2018

Uh oh!

allanrenucci commented Mar 1, 2018

Uh oh!

expz commented Mar 3, 2018

Uh oh!

Blaisorblade commented Mar 4, 2018

Uh oh!

allanrenucci commented Mar 4, 2018

Uh oh!

expz commented Mar 17, 2018

Uh oh!

allanrenucci commented Mar 18, 2018

Uh oh!

java.lang.NullPointerException on pattern match in dotty REPL #4051

java.lang.NullPointerException on pattern match in dotty REPL #4051

Comments

prayagupa commented Feb 28, 2018

Blaisorblade commented Feb 28, 2018

Uh oh!

allanrenucci commented Mar 1, 2018

Uh oh!

expz commented Mar 3, 2018

Uh oh!

Blaisorblade commented Mar 4, 2018

Uh oh!

allanrenucci commented Mar 4, 2018

Uh oh!

expz commented Mar 17, 2018

Uh oh!

allanrenucci commented Mar 18, 2018

Uh oh!