Experimental Streams using quotes #4317

biboudis · 2018-04-13T16:04:19Z

Port of the strymonas library as described in O. Kiselyov et al., Stream fusion, to completeness (POPL 2017) using Dotty's staging facility.

Ideas:

design a cleaner API, possibly investigate if it could be part of collections
experiment with alternative code generation schemes, e.g., generate fused code run wtih OpenCL/CUDA bindings
multidimensional types

biboudis · 2018-04-19T17:55:57Z

tests/run-with-compiler-custom-args/staged-streams_1.scala

+              producer.init(st => '{
+                Var('{ (_: Unit) => ()}){ advf => {
+                  Var('{ true }) { hasNext => {
+                    // Var('{ null.asInstanceOf[A] }) { curr => {


I would like to generate the following line:

var curr: A = null.asInstanceOf[A]

so I guess the only way is:

Var('{ null.asInstanceOf[A] }) { curr => ....

The minimal standalone example that corresponds to the code above is:

trait Bar[T: Type] { def meth(): Unit } class Foo[T: Type] { new Bar[T] { def meth() = { Var('(null.asInstanceOf[T])) { curr => '() } } } }

The problem is that I get the following error message and I don't know if thats a bug or I am doing something horribly wrong.

25 | new Bar[T] { | ^^^^^^ | constructor Bar in trait Bar does not take parameters

ping @nicolasstucki

trait Bar { def meth(): Unit } class Foo[T: Type] { new Bar { def meth() = { Var('(null.asInstanceOf[T])) { curr => '() } } } }

Var('(null.asInstanceOf[T])) { curr => '() } | ^ | access to Foo.this from wrong staging level: | - the definition is at level 0, | - but the access is at level 1.

Opened #4350 after investigation

biboudis · 2018-05-02T17:05:09Z

// Fixed: Note to self: the call to adv is not generated correctly in the nested case. Have to investigate

biboudis · 2018-05-03T17:32:24Z

@nicolasstucki parallel tests failed. I will restart the build.

nicolasstucki · 2018-05-04T15:45:43Z

tests/run-with-compiler-custom-args/staged-streams_1.scala

+  }
+
+  object Stream {
+    def of[A: Type](arr: Expr[Array[A]]): Stream[A] = {


Try changing it to

def of[A: Type: Liftable](arr0: Array[A]): Stream[A] = { val arr = arr0.toExpr ...

You will need some lifter for Arrays from the library.

nicolasstucki · 2018-05-04T15:47:56Z

tests/run-with-compiler-custom-args/staged-streams_1.scala

+     }
+
+    /** A stream containing the first `n` elements of this stream. */
+    def take(n: Expr[Int]): Stream[A] = Stream(takeRaw[Expr[A]](n, stream))


def take(n: Int): Stream[A] = Stream(takeRaw[Expr[A]](n.toExpr, stream))

@nicolasstucki what’s the advantage of your proposal? At first sight it seems less powerful.

The aim is to hide the Expr from the end user.

Then do we want fold to have the following signature?

def fold[W : Type](z: W, f: (W, A) => W): W

I’m not sure this is what we want?

You are right, it is not exactly what I want. We need to hide the Expr behind a macro. I will test my signature idea and then document it here if it works

nicolasstucki

I think we should merge like this. It will serve a great regression test. We should refine it further based on this work afterward.

biboudis · 2018-05-18T09:52:23Z

I have the changes you proposed ready but I was stuck in something during the refactoring. I will post here, maybe discuss and merge.

Thanks @nicolasstucki

julienrf · 2018-05-22T08:12:10Z

tests/run-with-compiler-custom-args/staged-streams_1.scala

+      * {{{
+      *           /* initialization for stream 1 */
+      *
+      *           var curr = null.asInstanceOf[Int]  // keeps each element from reified stream


This should probably be 0 instead of null. Also, it seems that curr is never updated.

👍

I was "updating" it as a comment. Fixing it to illustrate it better!

julienrf · 2018-05-22T08:18:06Z

tests/run-with-compiler-custom-args/staged-streams_1.scala

+              producer.init(st =>
+                Var('{ (_: Unit) => ()}){ nadv => {
+                  Var('{ true }) { hasNext => {
+                     Var('{ null.asInstanceOf[A] }) { curr => '{


I’m wondering if this can cause subtle bugs, since null.asInstanceOf[Int] produces 0, which is undistinguishable from the value 0 (by contrast with references, where null can always be distinguished from an object reference).

Why don’t you use Option[A] instead?

julienrf · 2018-05-22T08:27:22Z

tests/run-with-compiler-custom-args/staged-streams_1.scala

+       Var(z) { s: Var[W] => '{
+           ~foldRaw[Expr[A]]((a: Expr[A]) => '{
+             ~s.update(f(s.get, a))
+           }, stream)


Is there any difference between '{ ~s.update(…) } and s.update(…)?

It is the same

Though I would not change it. This kind of small additions have been useful to find bugs in the implementation. I would keep it like this for as a regression test.

However, nice catch @julienrf!

julienrf

This looks interesting! Do you have more insights on the generated code or on performance?

julienrf · 2018-05-22T08:47:36Z

tests/run-with-compiler-custom-args/staged-streams_1.scala

+       }
+     }
+
+     private def foldRaw[A](consumer: A => Expr[Unit], stream: StagedStream[A]): Expr[Unit] = {


It seems that we don’t need to take stream as a parameter since it is already a parameter of the Stream class. (btw name shadowing is dangerous!)

biboudis · 2018-05-28T11:50:20Z

tests/run-with-compiler-custom-args/staged-streams_1.scala

+
+            def init(k: St => Expr[Unit]): Expr[Unit] = {
+              producer.init(st =>
+                Var('{ (_: Unit) => ()}){ nadv => {


@nicolasstucki this is the line that triggers the exception after the last rebase

Fix in #4592

LPTK · 2018-06-13T16:09:44Z

@biboudis very cool! Has this implementation been benchmarked? Does it expand full stream pipelines into low-level loops, as in the paper?

biboudis · 2018-06-13T17:35:44Z

Thanks @LPTK. Yes, by inspection, the code generates the expected imperative low-level loops. If needed we will benchmark it in the future. For now this is merely a big test case for us.

biboudis added stat:wip area:quoting labels Apr 13, 2018

biboudis commented Apr 19, 2018

View reviewed changes

biboudis force-pushed the streams-dev branch 4 times, most recently from 55b5913 to 087a191 Compare April 29, 2018 15:45

biboudis force-pushed the streams-dev branch from efc5484 to 8fa59a6 Compare May 3, 2018 15:11

biboudis requested review from OlivierBlanvillain and nicolasstucki May 3, 2018 15:17

biboudis removed the stat:wip label May 3, 2018

biboudis requested a review from odersky May 4, 2018 09:09

nicolasstucki reviewed May 4, 2018

View reviewed changes

biboudis requested a review from julienrf May 9, 2018 11:57

nicolasstucki approved these changes May 17, 2018

View reviewed changes

nicolasstucki assigned biboudis May 17, 2018

julienrf reviewed May 22, 2018

View reviewed changes

biboudis force-pushed the streams-dev branch from 1a637b5 to 6923432 Compare May 28, 2018 10:08

biboudis commented May 28, 2018

View reviewed changes

biboudis force-pushed the streams-dev branch from 6fa33a0 to 2599e06 Compare May 28, 2018 12:46

Fix scala#4591: Unpickle type quotes with TreeReader.readTpt

83cef51

biboudis force-pushed the streams-dev branch from 2599e06 to 6fa33a0 Compare May 29, 2018 08:03

biboudis added 25 commits May 29, 2018 10:21

Enable fold

be16938

Implement fold

ff68092

Implement map

99071e0

Implement flatMap

58e1726

Implement filter

83f8ed2

Comment out implementation of take

7bfbb06

Implement take

ad06ab0

Move to custom args for run-with-compiler

a2e4c49

Implement zip (for linear streams)

0328028

Implement zip (linear/nested)

9748f0d

Implement zip (nested/linear)

6dfbe9d

Refactor and add comments

bdfdb75

WIP implemention of reifying nested streams for zip

d565b1b

Add Type annotations

01058cc

Progress in makeLinear

395bd6d

Fix types for zip

4027fa2

Compile OK

d38fc83

Rebase and adjust type signatures

0d02cea

Fix Types

070c8a2

Implement zip (nested/nested)

e2e9484

Add comments

ad2f6c8

Bug fix and more comments

48ad346

Fix nested zip bug

e949a8a

Add note about the port

1d8b789

Update comment to reflect curr changing

3c46da8

biboudis force-pushed the streams-dev branch from 6fa33a0 to 3c46da8 Compare May 29, 2018 08:22

biboudis merged commit 78a5e09 into scala:master Jun 13, 2018

allanrenucci deleted the streams-dev branch June 13, 2018 14:21

Experimental Streams using quotes #4317

Experimental Streams using quotes #4317

Uh oh!

Conversation

biboudis commented Apr 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

biboudis Apr 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

biboudis commented May 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

biboudis commented May 3, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julienrf May 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicolasstucki left a comment

Choose a reason for hiding this comment

Uh oh!

biboudis commented May 18, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

biboudis May 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julienrf left a comment

Choose a reason for hiding this comment

Uh oh!

julienrf May 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LPTK commented Jun 13, 2018

Uh oh!

biboudis commented Jun 13, 2018

Uh oh!

Uh oh!

biboudis commented Apr 13, 2018 •

edited

Loading

biboudis Apr 19, 2018 •

edited

Loading

biboudis commented May 2, 2018 •

edited

Loading

julienrf May 22, 2018 •

edited

Loading

biboudis May 28, 2018 •

edited

Loading

julienrf May 22, 2018 •

edited

Loading