Add dottydoc #1453

felixmulder · 2016-08-15T17:35:45Z

Dottydoc - GSoC

In all ways and means dottydoc is a compiler. It uses the dotty frontend phase to do parsing and typing - after this the documentation phases take over. Similarly to dotty, dottydoc uses fused phases as a way of transforming the documentation AST.

The new scaladoc alias dottydoc is comprised of two parts: a documentation hook and a front end hook.

There are still a few missing features to dottydoc - and this PR was opened now due to the GSoC project coming to an end as well as the limitations of the CI (100 commits - will it pass the tests!?).

The missing parts are currently AFAIK:

Grouping
Usecase

Scaladoc has a long history, and as such features may have been overlooked, the missing parts as listed will be implemented in the coming weeks together with any missing features reported and vital.

Architecture

                         source files
                               |
                               v
                       +--------------+
 (template) -------->  |              |
                       |   dottydoc   | -----> JSON/html
[resources] -------->  |              |
                       +--------------+

(...) == optional file
[...] == optional list of files

Because dotty needs to bootstrap, we cannot have the client UI be part of the repository if we want to use external scala dependencies. This obstacle became an incentive to do something better - and the idea of third party templates was born. Currently dottydoc supports a specific frontend format which enables users to replace the frontend if they so choose (maybe they don't like the defaults, maybe they can do better - diversity is good).

To build your own frontend this is what you'll need:

.
├── template.html
├── resource1.js
└── resource2.css

The templating engine supported is currently mustache

The template will be given an instance of Entity for each top-level object (traits, objects, classes, packages). Entity has the following signature:

trait Entity {
  def name: String
  /** Path from root, i.e. `scala.Option$` */
  def path: List[String]
  def comment: Option[Comment]
  def kind: String
  def parent: Entity
}

The template is then used to create a collection of static HTML pages à la the old scaladoc. Each template is also given the path to assets (directory) and index (javascript file). As such it is quite easy to build your own template. Example:

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>Dottydoc - {{name}}</title>
        <script type="text/javascript" src="{{index}}"></script>
        <script type="text/javascript" src="{{assets}}/dottydoc.js"></script>
        <link rel="stylesheet" href="{{assets}}/index.css">
    </head>
    <body>
        <div id="main-container">
            <h1>{{name}}</h1>
        </div>
        <script>
            (function() {
                Index.currentEntity = {{&currentEntity}};

                var addMember = function(entry) {
                    document.getElementById("main-container").innerHTML += entry.name + "</br>";
                };

                if (Index.currentEntity.members !== undefined)
                    Index.currentEntity.members.forEach(addMember);
            })();
        </script>
    </body>
</html>

The Index object will look like this after the template has been written:

{
    "packages": {
        "scala": { ... },
        "scala.collection": { ... },
        ...
    }
    "currentEntity": {
        "name": ...
    }
}

There is of course already a default front end in place dottydoc-client. This one will be used by default by sbt. This project is available here: https://github.com/lampepfl/dottydoc-client

To view the documentation rendered by the 100th commit in this PR: https://dl.dropboxusercontent.com/u/358427/gsoc-dottydoc/docs/scala/Option.html

There are parts in dottydoc that have been developed for the previous scaladoc and adapted to dottydoc most prominently by @odersky, @VladUreche @manojo - thank you for your hard work.

There is still a ton to do and I will make a small list here of things I plan to work on, and that I'd be happy to receive help on.

The great aftermath of the dottydoc PR

@usecase
@group
Better search that displays members (client)
Rename docs -> API and add support for documentation as reference
Prototype markdown in doc-comments
Properly handle synthetically added methods (argument names are mangled)

xeno-by · 2016-08-16T19:23:15Z

This is an impressive contribution. Congratulations!

There have always been two things about scaladoc that I thought could be improved. I wonder if you could comment on whether they can be potentially addressed in dottydoc.

Incremental compilation. If I change just one file in the project, I'd like to also regenerate at most one page in the documentation. At the moment, if I'm not mistaken, scaladoc needs a full recompilation. Can something be done about that?
Separate compilation. If I want to include N projects in my documentation, then I need to compile all of them in one compilation run. I think this is pretty inconvenient, and also not always feasible is macros are in play. What do you think about this one?

felixmulder · 2016-08-16T20:01:23Z

Thank you @xeno-by, much appreciated.

Incremental compilation. If I change just one file in the project, I'd like to also regenerate at most one page in the documentation. At the moment, if I'm not mistaken, scaladoc needs a full recompilation. Can something be done about that?

It is not something that has been taken into consideration thus far - but it is definitely something worth investing in further down the line.

Furthermore, I do believe it is possible to preserve the index and mutate it between runs - the API could be made to handle this in some convenient way that works well with the build tools.

Separate compilation. If I want to include N projects in my documentation, then I need to compile all of them in one compilation run. I think this is pretty inconvenient, and also not always feasible is macros are in play. What do you think about this one?

My first reaction is that this could/should be solved using TASTY - but I'm not currently 100% confident in what has been included in the spec (i.e - does TASTY serialize comments attached to their corresponding entity?).

An other alternative would perhaps be to define some form of lookup using the scaladex - but perhaps the tools are better kept separate.

VladUreche · 2016-08-16T22:27:48Z

It is not something that has been taken into consideration thus far - but it is definitely something worth investing in further down the line.

The problem is changes are not local but ripple out -- e.g. adding a new class means its superclasses get another subclass listed on their pages.

retronym · 2016-08-17T05:29:49Z

The problem is changes are not local but ripple out -- e.g. adding a new class means its superclasses get another subclass listed on their pages.

It might be useful to think about this in terms of a compiler + linker. The compiler extracts information from a single .scala file into a corresponding output file (perhaps attributes in the .class file?). References to external data should be expressed symbolically. This could be resolved by a linker (that generates the entire site). The linker itself could be incremental in manner of ScalaJS's linker (ie: "I know that I need to rebuild this section of the page in all superclasses"). The "linker" might even be implemented as Javascript in the view itself.

felixmulder · 2016-08-19T13:33:05Z

/rebuild

This commit adds return values to defs and vals - but does not provide a link to these. This is to be continued

…serializing to json

To access `DocBase`: `ctx.docbase.docstring(...)`

…method

…mplate for html docs

…mbers

smarter · 2016-08-21T00:13:10Z

dottydoc/src/dotty/tools/dottydoc/api/scala/Dottydoc.scala

+ * The tool can also generate JSON from the created index using "indexToJson"
+ * or directly using "createJsonIndex"
+ */
+trait Dottydoc extends DocDriver {


Why do you have both Dottydoc.java and Dottydoc.scala ?

The idea is to have a Java interface for when the time comes that releases are compiled with dotty. As such, tools compiled with dotty should be able to use the Scala interface, while incompatible projects (such as sbt on 2.10) will still be able to use the Java interface.

OK, should this interface be part of dotty-interfaces ?

I'm okay with that as long as it's on the same classpath available to the bridge as dotty 😄

Let me know and I'll make it happen.

OK, let's keep it as is for now

odersky · 2016-08-23T14:44:44Z

Any objection to merge this now?

felixmulder added In progress and removed In progress labels Aug 15, 2016

felixmulder added 22 commits August 19, 2016 15:37

Add project and impl for initial barebones dottydoc

8d8d87b

Port cooking of strings from NSC

1b20568

Add module member lookup

9d4df20

Fix compilation for multiple sources

6bd4969

Add summary comment, cleanup

36e63e8

Add command alias for dottydoc

fb6c152

Minify css

980a662

Improve layout with search and replace confusing Packages title

02a12f9

Add the scaladoc CLI args that make sense for dottydoc

2de658d

Add recursive CLI args, move other doc args

887759e

Fix package list links

46ac1bf

Add initial return values to members - to be updated

32b4a9c

This commit adds return values to defs and vals - but does not provide a link to these. This is to be continued

Fix packages not ending up in packages map in Phases.scala

b480bd1

Move dynamically generated content to JVM side

cbc05e0

Move member rendering back to client (for event listeners)

80a9077

Split implementation to internal mutable and external interface

db659dc

Filter out companion objects from package-view

0fb1058

Add companions to package view w ability to choose between them

86f816d

Adjust font configuration for finalized package view

53fc2c0

Filter out ProductX when X > 3 in package view

cb42a04

Break out commentCache to CommentParser

80b027f

Better prints while documenting packages

60b5dc3

felixmulder added 20 commits August 19, 2016 15:37

Sort members when serializing to JSON

c4ec1b6

Add a SortMembers phase to sort the members instead of doing it when …

87ecca2

…serializing to json

Remove shared directory - nothing shared anymore

553f9b2

Fix #25: move doc related structures to DocBase

98b61dd

To access `DocBase`: `ctx.docbase.docstring(...)`

Get docstring from overriden symbol if not present on the overriding …

94dd0bc

…method

Add info on where implicitly added members originate from

922788a

Fix phase fusion for subpackages, fix links in implicitly added methods

5fdc353

Fix type params of classes, traits defs, which broke after new HK scheme

20fda57

Implement annotation parsing like CommentFactoryBase from nsc

079e3db

Remove client from dottydoc - no more Scala.JS deps!

5cd1d51

Add currentEntity to templating API

0d89be8

Add scaladoc interface stub

b1defa0

Implement bridge that generates json on doc command

a252074

Implement working docs requiring manually specifying resources and te…

bd77519

…mplate for html docs

Implement dotty bridge for doc

d6b5c3e

Add documentation to dottydoc API

abf1632

Move client test to client repo

0a9003d

Add snapshots resolver

9879b90

Properly materialize supertype links

59c6c0d

Add ImplicitlyAddedEntity as a common trait for implicitly added me…

d8c02ec

…mbers

felixmulder force-pushed the topic/dottydoc branch from 2ad1877 to 556c48e Compare August 19, 2016 13:37

smarter reviewed Aug 21, 2016
View reviewed changes

Add phase to deal with constructors

0b69be6

felixmulder force-pushed the topic/dottydoc branch from 556c48e to 0b69be6 Compare August 21, 2016 13:52

odersky merged commit 265ade0 into scala:master Aug 23, 2016

odersky removed the Needs review label Aug 23, 2016

OlivierBlanvillain pushed a commit to OlivierBlanvillain/dotty that referenced this pull request Dec 12, 2016

Fixed scala#1453.

2ca63a7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add dottydoc #1453

Add dottydoc #1453

Uh oh!

felixmulder commented Aug 15, 2016 •

edited

Loading

Uh oh!

xeno-by commented Aug 16, 2016

Uh oh!

felixmulder commented Aug 16, 2016

Uh oh!

VladUreche commented Aug 16, 2016

Uh oh!

retronym commented Aug 17, 2016

Uh oh!

felixmulder commented Aug 19, 2016

Uh oh!

smarter Aug 21, 2016

Uh oh!

felixmulder Aug 21, 2016

Uh oh!

smarter Aug 21, 2016

Uh oh!

felixmulder Aug 22, 2016 •

edited

Loading

Uh oh!

smarter Aug 22, 2016

Uh oh!

odersky commented Aug 23, 2016

Uh oh!

Uh oh!

Add dottydoc #1453

Add dottydoc #1453

Uh oh!

Conversation

felixmulder commented Aug 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dottydoc - GSoC

Architecture

The great aftermath of the dottydoc PR

Uh oh!

xeno-by commented Aug 16, 2016

Uh oh!

felixmulder commented Aug 16, 2016

Uh oh!

VladUreche commented Aug 16, 2016

Uh oh!

retronym commented Aug 17, 2016

Uh oh!

felixmulder commented Aug 19, 2016

Uh oh!

smarter Aug 21, 2016

Choose a reason for hiding this comment

Uh oh!

felixmulder Aug 21, 2016

Choose a reason for hiding this comment

Uh oh!

smarter Aug 21, 2016

Choose a reason for hiding this comment

Uh oh!

felixmulder Aug 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smarter Aug 22, 2016

Choose a reason for hiding this comment

Uh oh!

odersky commented Aug 23, 2016

Uh oh!

Uh oh!

felixmulder commented Aug 15, 2016 •

edited

Loading

felixmulder Aug 22, 2016 •

edited

Loading