diff --git a/minutes/_posts/2018-12-05-december-5-2018.md b/minutes/_posts/2018-12-05-december-5-2018.md new file mode 100644 index 0000000..3f63515 --- /dev/null +++ b/minutes/_posts/2018-12-05-december-5-2018.md @@ -0,0 +1,392 @@ +--- +layout: contact +--- + +# Minutes of the 11th meeting of the Scala Center, Q4 2018 + +Minutes are [archived](https://scala.epfl.ch/records.html) on the +Scala Center website. + +## Summary + +The following agenda was distributed to attendees: +[agenda](https://github.com/scalacenter/advisoryboard/blob/master/agendas/011-2018-q4.md). + +Spotify has joined the advisory board. Julien Tournay is +their representative. + +After this meeting, Lars Hupel is stepping down as community +representative. A replacement has yet to be nominated. + +Scala Center activities for the past quarter focused on +Coursier, Almond (formerly Jupyter-Scala), Bloop and BSP, +Zinc, build pipelining, 2.13 collections, Dotty, Metals, +Scalameta, Scalafix, Scala.js, SIP meetings, MOOCs, +Scala Days 2019, and ScalaIO. + +Full details on these activities are in +[Sébastien's report](https://docs.google.com/document/d/1MwzPQtYIaJ4_5z7-StJI-gKiY-oV_6Or7PEvnJBR92k/edit?usp=sharing). + +Other topics discussed at the meeting included the Center's +sponsorship terms. + +Two new proposals were made: + +* [SCP-019](https://github.com/scalacenter/advisoryboard/blob/master/proposals/019-scala-214-30-back-compat.md): Focusing on backwards compatibility for Scala 2.14 and Scala 3.0 (Twitter) + * voted on, not accepted +* [SCP-020](https://github.com/scalacenter/advisoryboard/blob/master/proposals/020-sbt-transitive-dependencies-conflicts.md): SBT transitive dependency conflicts management improvements (Spotify) + * voted on and accepted + +SCP-20 was exceptionally accepted as a late submission +from Spotify, as they were only confirmed as board members +between the submission deadline and the meeting. + +## Date, Time and Location + +The meeting took place virtually, via Google Meet, on Wednesday, +December 5, 2018 at 3:30pm (GMT). + +Minutes were taken by Seth Tisue (secretary). + +## Attendees + +Board members present: + +- James Belsey, Morgan Stanley +- Eugene Burmako and Stu Hood, Twitter +- Lars Hupel, community/Typelevel +- Olga Makhasoeva, 47 Degrees +- Adriaan Moors, Lightbend +- Jonathan Perry, Goldman Sachs +- Frederick Reiss, IBM +- Julien Tournay, Spotify +- Mark Waks (a.k.a. Justin du Coeur), community/Artima, on behalf of Bill Venners + +Apologies: + +- Thomas Gawlitza, SAP + +Officers: + +- Sébastien Doeraene (director), EPFL +- Jon Pretty (chairperson), Propensive +- Martin Odersky (technical advisor), EPFL +- Seth Tisue (secretary), Lightbend + +## Proceedings + +As chairperson, Jon Pretty conducted the meeting. + +He welcomed Spotify as the newest advisory board member, and Julien +Tournay as their representative. Julien works for Spotify in Sweden. + +Jon also welcomed Olga Makhasoeva as 47 Degrees' new representative +on the board. + +Filling in for Bill Venners at this meeting was Mark Waks, Bill's +colleague at Artima. + +### Activities + +As the Center's Executive Director, Sébastien Doeraene summarized +Scala Center activities since the last meeting. + +Most of Sébastien's remarks were based on his +[detailed report](https://docs.google.com/document/d/1MwzPQtYIaJ4_5z7-StJI-gKiY-oV_6Or7PEvnJBR92k/edit?usp=sharing) +on the Center's recent activities. + +The following notes are a supplement to Sébastien's report: + +Coursier isn't itself new, but it is a new project for the Scala +Center. Alexandre Archambault, who is now a full-time engineer at the +Scala Center, is Coursier's original open-source developer. Seb +highlighted Alex's plan to add publishing support to Coursier and +eventually integrate Coursier into sbt more directly as a faster +replacement for Ivy. "It's a big enabler" of productivity, "basic +speeds of day-to-day tools" that Scala developers use. A new +documentation site for Coursier will be released "soon". + +Martin Duhemm has left the Scala Center. The Dotty IDE work described +in Seb's report was Martin's last project. + +Seb's report covers two different IDE projects, namely Dotty IDE and +Metals. Unlike Dotty IDE, Metals aims to support Scala 2. The two +projects' features and UI "will be reconciled in the coming months," +says Seb. Metals "doesn't have a lot of features" yet but "the +features it has are really polished" and "it's really simple to +install and to import projects." The goal is to add features +gradually, "to do one thing at a time but do it really well." + +Jon asked what quality level that Metals has attained so far. Seb +said that the handling of compiler diagnostics is already "perfect". +The go-to-definition feature has good test coverage, but users may +still uncover unknown bugs. Importing builds, via Bloop, may fail on +especially complex builds. Julien offered that Metals "worked +perfectly" so far on a 20-module project he's been trying it with, +though it "takes a while" to initialize. + +Jon asked about Scala Days sponsorships. Seb said that interested +sponsors should get in touch with Darja at the Scala Center. + +Eugene asked what's next for build pipelining. Seb said that Jorge +will continue to work on it, next steps in that work include support +for incremental compilation and mixed Scala/Java codebases. "Once +it's usable for incremental compilation we'll make a bigger fuss about +it and advertise it much more widely," said Seb. + +Eugene also asked about the status of Dotty IDE. Seb returned to the +question of aligning the work on Dotty IDE with the work on Metals. +In many ways they are similar (for example, they are both LSP based +and both integrate with VS Code and other editors). But right now the +user must install "one or the other", installing both tends to "not +quite work." These projects need to become "one concerted effort" +with a single unified download, which would then operate in either +Scala 2 or Dotty mode depending on what kind of project you have. + +Stu asked about the status of +[SCP-018](https://github.com/scalacenter/advisoryboard/blob/master/proposals/018-converging-214-30.md) +("Converging the Intermediate Representation of Scala 2.14 and Scala +3.0"), "was anyone assigned to work on that yet?" Seb said the Center +is looking for "a new person we could assign this project to." (If a +new person can't start soon, someone else might be assigned in the +meantime.) + +### Financial report + +"We got the final numbers for the MOOC income for Q1 through Q3 of +2018," said Seb. This totaled just under 300K CHF, of which the +Center's share is 215K CHF. (The CHF/USD exchange rate is currently +1-to-1.) This is similar to past proceeds, so that part of the budget +is "stable". + +With Spotify joining the board, there are now eight member companies. +There may be enough budget to open an additional position, besides the +one vacated by Martin Duhemm, but it's not certain yet. + +Jon asked if there was any possibility of using extra money to +increase salaries, to help ensure retention of engineers. Seb said +that EPFL's rules around compensation are so strict that there's +really no leeway. + +Jon also asked if the Scala Days 2019 (in Lausanne) has any financial +risks or financial opportunities for the Center. Seb said that "the +finances of Scala Days are a completely separate budget". Because +there is only one Scala Days in 2019, a larger hall was obtained, on +the assumption that attendance will be higher than two separate +conferences on either side of the Atlantic. The break-even point is +900 attendees, which is not very many more than attended Scala Days +Berlin in 2018. + +### Proposals + +Both proposals were presented discussed before either proposal was voted on. + +#### SCP-019: Focusing on backwards compatibility for Scala 2.14 and Scala 3.0 + +Proposed by Stu Hood on behalf of Twitter. + +The theme of the proposal is the relative priority, during the transition +of Scala 3, of backward compatibility (Scala 3's ability to consume +libraries built using Scala 2) versus forward compatibility (Scala 2's +ability to consume libraries built using Scala 3). + +See the [proposal text](https://github.com/scalacenter/advisoryboard/blob/master/proposals/019-scala-214-30-back-compat.md) +for details. + +There was some discussion on the [pull request for the +proposal](https://github.com/scalacenter/advisoryboard/pull/44) + +between Martin and Stu. This resulted in some [revisions to the +proposal](https://github.com/scalacenter/advisoryboard/pull/47) +shortly before the meeting. These changes are included in the +proposal text linked above, and the updated version is also what the +board finally voted on. + +In presenting the proposal, Stu noted that an alternative to forward +compatibility already exists, namely cross-compilation. "Libraries +can do what they already do," namely cross-compile and cross-publish, +in this case against both Scala 2 and 3, as long as they only use the +common subset of the language (which encompasses nearly everything in +Scala 2). + +"This is not an anti-TASTy proposal", Stu said. "This is a proposal +to use whatever means necessary to allow backwards compatibility." If +Scala 3's support for Scala 2's pickle format is good enough to get +good backward compatibility, then so be it. But if Scala 2 emitting +TASTy turns out to be a better way to achieve backward compat, then so +be it also. (Whereas Scala 2.14's proposed ability to _consume_ TASTy +is less relevant to backwards compat.) + +Martin commented on the proposal. He described Dotty's already +existing Scala 2 pickle support as "a solved problem." "Every Dotty +program ever written uses Scala 2 pickle support" and it's never been +a problem, so Martin hopes that the backward compatibility problem is +actually already solved, but does that still need to be validated +further? To more fully validate it, we need to test it with "a lot of +code". There are two ways to get that much code, one is to wait for +the community to catch up and write a lot of Scala 3 code, but that +takes a long time. Or you port a lot of existing Scala 2 code, but +that takes effort, especially for porting macros, but also (for +example) if the code hits differences in type inference. The Scala +Center can help Scala 2 projects port their code when the time comes, +but in the meantime there isn't anything to do to improve binary +compatibility, so Martin sees no need to do deprioritize the +forward-compatibility parts of SCP-018. + +Jon asked Adriaan what the impact on Lightbend's 2.14 efforts would +be. Adriaan addressed that and also commented more generally on the +new proposal. "The key point is that forward compatibility is also +important" for Scala 3 "because in order to identify potential +challenges with Scala 3, we need to be able to link both ways. +Applications might not upgrade to Scala 3 right away, but libraries +may be eager to upgrade." + +Adriaan also said that SCP-018 was intentionally "pretty vague" about +the exact nature of the best path forward. "It's the task of the Scala +Center" to figure out exactly what to do, depending on what engineers +are available, and any other factors that arise as the work progresses. +Seb agreed that this new proposal (SCP-019) "is directing, more than +usual, what the Center should do and how we should do it." + +Eugene asked what barriers exist, or are expected to exist, for +cross-compiling libraries against Scala 2 and Scala 3. Understanding +that better could help identify the best approach to compatibility. +Stu added that the Dotty community build is currently "leaf nodes", +that is, libraries with few or no dependencies. We won't know more +about compatibility until it's expanded to cover inter-library +dependencies better. And Jon observed that we need to test +applications that depend on the libraries, too. + +Martin says the blocker, currently, for expanding the scope of the +Dotty community build is macros. "Transitive dependencies using +macros are absolutely everywhere as you go up the stack. That's +why it's so difficult. Bottom dependencies need to be adapted to +use new-style macros" before we can test downstream projects. +So yes "there are porting problems", but "they are not related to +binary compatibility. Macros are the problem." But if the Scala +Center can identify key macro-based libraries and help port them, +that will be a "big win". + +Mark added that ScalaTest is currently being ported to Dotty, so that +will be a "big, chewy test" of a widely-used library that is +macro-based. He also mentioned that we need to keep in mind that the +world of Scala development at companies is very different than what we +see in the open-source world. For example, relatively few companies +are writing their own macros, even though open-source macro-based +libraries are widely used everywhere. So we'll need to keep that in +mind when gathering data on compatibility. + +Martin agrees with Stu that producing a "compatibility reference" is +needed, so that the community has the information it needs to +cross-compile and/or cross-consume between Scala 2 and 3, but he sees +producing this as primarily LAMP's and Lightbend's job, rather than +primarily the Scala Center's job. + +Seb explained some the motivation for valuing forward compatibility, +as he sees it, though "not taking sides". He explained, "If we only +support 3 depends on 2, but not the other way around, then library +developers are stuck on 2 for the next 5 years, since if their +codebase must cross-compile, they cannot benefit from Scala 3's +features. That means that in practice, the visible Scala ecosystem +would remain on Scala 2 for a long time, and that could send a +worrying message to the public." Martin seconded this. He said most +of the users he talks to who are worried about Scala 3 are worried +about slow adoption, but then the forward-compatibility part of +SCP-018 strongly reassures them. + +Some complex discussion about macros and the overall compatibility +followed, difficult to summarize. The key point is that neither +forward compatibility nor backwards compatibility provide macros +that work across both Scala 2 and Scala 3. + +It's clear that library authors and users will need guidance to +understand what will or won't work in what scenarios, and when it +matters how macros are involved. + +Jon noted that even if SCP-019 isn't accepted, that doesn't close the +door on other future proposals aimed at refining SCP-018, as the +overall Scala 3 situation develops further. + +**Voting**: three in favor, four opposed (with Seb casting the +tie-breaking vote, and promising to "take the proposal into account" +regardless), three abstentions, one member not present. The proposal +is not accepted. + +#### SCP-020: SBT transitive dependency conflicts management improvements + +Proposed by Julien Tournay on behalf of Spotify. + +See the [proposal text](https://github.com/scalacenter/advisoryboard/blob/master/proposals/020-sbt-transitive-dependencies-conflicts.md) +for details. + +In presenting the proposal, Julien said that library version conflicts +have been a recurring problem in deploying code at Spotify. Sometimes +"we only realize there is a conflict when we run our data pipelines," +sometimes not until after the job has been running for hours. + +Lars asked to what extent the proposed work on sbt should overlap or +interact with the work the Center is now doing on Coursier. + +Julien said that Coursier isn't the default in sbt, so for now, "most +builds" would have the same issue even if it were fixed in Coursier. +But if Coursier fixes it and also becomes the de facto standard for Scala +builds, then yes, "done." + +Seb observed that sbt does already print eviction warnings. Lars +asked how a build tool can do better than that, given the lack of an +accepted standard way for libraries to signal binary +compatibility. Not all libraries use semantic versioning, or do it the +same way. "The reason I sound pessimistic," said Lars, "is that I'm +concerned about false positives," a tool reporting conflicts where no +incompatibility exists. But the proposals seems to give "leeway" to +the Scala Center to do whatever's feasible and helpful. + +Seb suggested perhaps adding a "2b", between levels 2 and 3 in the +proposal, which would involve "a more elaborate API for conflict +resolution", since `ConflictManager`, even if it worked perfectly as +designed, arguably isn't a flexible enough mechanism, for the same +reasons that Lars brought up: "you want to resolve conflicts in a +different way for different libraries." + +Mark spoke up in support of the proposal, because the problem is real, +and the proposal gives the Scala Center flexibility to choose how to +address it. "At least level 1 would totally be helpful." Perhaps +plugins such as sbt-dependency-graph could become part of sbt. + +Jonathan and James both expressed some doubt about the usefulness, in +their shops at least, of upping the overall level of "intricacy" +around dependencies, especially in polyglot environments, by +introducing yet more conflict resolution rules and mechanisms. + +**Voting**: seven in favor, one opposed, one abstention, +one member not present. The proposal is accepted. + +### Community feedback + +Lars Hupel announced that he is stepping down as community +representative; Jon thanked him for his more than two years of +service. A replacement has yet to be nominated. (Last time around, +the spot was offered to Typelevel to fill; Jon anticipated that +happening again, but it's up to the board.) + +Mark said that "he's glad to see a focus on tooling, since that is still +the thing that comes up most often" in his conversations with community +members. + +### Other business + +At the previous meeting, there was some discussion about perhaps +changing the Center's sponsorship terms. Jon asked Seb if this has +been considered further yet, and also asked if it would be good to +create a concept of a "developer sponsor" who would pay a smaller fee +but make up for it by contributing development work directly. But the +board agreed there wasn't really time to discuss it. James asked if +the Center could make specific suggestions, to be voted on at the next +meeting. Seb agreed to raise the topic on the board's mailing list, +for discussion in advance of the next meeting. + +## Conclusion + +Jon said that as usual, the next meeting will be in approximately +three months, perhaps early march. Then the meeting after that will +be held in Lausanne, in conjunction with Scala Days.