Introducing transform plugins #499

etpinard · 2016-05-03T16:52:35Z

@mdtusz @chriddyp @bpostlethwaite @alexcjohnson @jackparmer

This PR adds support for custom data transforms to be plugged into plotly.js via Plotly.register.

In brief

Plotly.register can now register transform modules 9ac6d0f
The supply-defaults step can now handle data transforms. IMPORTANT User data and gd._fullData no longer have to be of the same length. 06f2e6d
Plotly.restyle can now handle transformed data 8ab9d0c

Examples

A filter and group-by were added as examples in 2697975

Does it work?

See 56665f2

TODOs

Agree on how to handle legend items in group-by transformed data.
Agree on the API for transform modules' transform method.
Agree on where to place and how to named full transformed data and other intermediate objects that may be useful for folks building toolpanels. (done, see 9a341c1)
Integrate transforms attributes into plot schema. (done, see b5ec01e)

- split per-trace defaults step from plots.supplyDataDefaults - call transform module supplyDefaults at end of trace-defaults routine, so that transform default logic can rely on trace defaults - add an applyTransforms step which generates 'expanded' data which themselves go through a trace-default step. - transform functions are expected to supply an array of traces

- the 'traces' array of trace index is now based off gd.data - make all transforms restyle calls go through doCalcdata which is the worse case scenario (i.e. we take no chance!)

etpinard · 2016-05-03T17:06:34Z

EDIT

This problem below is solved via 9a341c1 where we made the expanded full traces carry the index of their parent gd.data trace in the index field (as for regular / non-expanded full traces).

More on the first TODO item ⏫ :

First, see example of group-by transform module here.

In this module, transformed trace are expanded into as many traces as there are groups items. At the moment, these expanded traces are all displayed in the legend. BUT, these cannot be as they do not match gd.data one-to-one and I made Plotly.restyle to be based off gd.data in 8ab9d0c.

How to resolve this problem? Here are a few possibilities:

Force expanded traces from the same user data trace to have to be toggled at once. Essentially, like they had the same legendgroup value.
Write the group-by transform plugins such that as groups and the number of traces sent to Plotly.plot match one-to-one. It would be easy to have the transform look up the same data arrays for the transform step. This is the most robust solution from plotly.js's perspective.
Make Plotly.restyle handle transformed indices too. Maybe with an option flag? But this sounds useful only for the visibility toggling.
Add an attribute to the group-by transform module to would toggle the visibility of a particular group and somehow let the legend code know about it. For example,

transforms: [{
  type: 'groupby',
  groups: ['a', 'b', 'a', 'b'],
  groupsVisible: ['a']
}]

would hide the 'b' group. Note that some people have asked for custom legend interactions (see #65). Maybe this isn't such a bad idea.

alexcjohnson · 2016-05-03T21:41:57Z

src/plots/plots.js

+        else {
+            dataOut.push(fullTrace);
+            fill(modules, fullTrace._module);
+            detect(trace);


detect(fullTrace); ?

in fact, this untransformed case needs i linked in as well. Might as well make eg:

function addFinalTrace(finalTrace) { finalTrace._input = trace; finalTrace._inputIndex = i; finalTrace._expandedIndex = cnt; dataOut.push(finalTrace); fill(modules, finalTrace); detect(finalTrace); cnt++; }

since all of that needs to happen identically whichever way you added a trace to _fullData.

Good catch ⚾ . Thanks!

chriddyp · 2016-05-03T23:40:59Z

this looks awesome! with this architecture, my "visual" transforms will also need to have something like a "transform inverse".

data

[{
    x: [1, 2, 1, 2],
    y: [3, 1, 5, 4],
    transforms: [{type: 'groupby', groups: ['a', 'a', 'b', 'b']}]
}]

_fullData

[
   {
      x: [1, 2], y: [3, 1], 
      // + all of the defaults
      marker: {color: 'blue', size: 6, symbol: 'circle', line: {}}
   },
   {
      x: [1, 2], y: [5, 4], 
      // + all of the defaults
      marker: {color: 'red', size: 6, symbol: 'circle', line: {}}
   }
]

transformInverse(data, _fullData, transformIndices)

[{
    x: [1, 2, 1, 2],
    y: [3, 1, 5, 4],
    transforms: [{
        type: 'groupby',
        groups: ['a', 'a', 'b', 'b'],

        // + all of the default properties mapped *back* to the original trace
        // the transform author writes this `transformInverse` function
        marker: {
           color: {
                  a: 'blue', b: 'red'
            },
            size: 6,
            symbol: 'circle',
            line: {}
        }
   }]
}]

my style panels will be built off of transformInverse(_fullData).

similar example with candlestick charts:

data

{
    x: [1, 2, 3],
    transform: [{
       type: 'candlestick',
       open: [6, 2, 3], high: [4, 1, 3], low: [3, 4, 1], close: [5, 8, 3]
    }]
}

_fullData

[
   // "high" trace 
   {
        x: [....], y: [....],
        type: 'box',
        // + all the default box properties
        orientation: 'v',
        jitter: 0,
       marker: {
           color: 'blue', 
           // ... and more
       },
       // .. etc
   // "low" trace 
   {
        x: [....], y: [....],
        type: 'box',
        // + all the default box properties
        orientation: 'v',
        jitter: 0,
       marker: {
           color: 'red', 
           // ... and more
       },
    }
]

transformInverse(data, _fullData, transformIndices)

{
    x: [1, 2, 3],
    transform: [{
       type: 'candlestick',
       open: [6, 2, 3], high: [4, 1, 3], low: [3, 4, 1], close: [5, 8, 3],

        // then, all the default properties mapped back
        orientation: 'v',
        jitter: 0,
        marker: {
            color: {high: 'blue', low: 'red'},
         }
    }]
}

jcheng5 · 2016-05-09T20:41:03Z

test/jasmine/assets/transforms/filter.js

+
+        if(k.charAt(0) === '_') return;
+
+        if(k === 'transforms') {


I think k === 'colorscales' might need a special case too. I'm not using this plugin but adapted this approach for my own purposes, and I found that when coloring by a continuous variable you end up with errors unless colorscales are left alone. (And just by inspecting the docs, perhaps ticktext and tickvals might be special as well.)

The ideal solution would use our Plotly.PlotSchema.crawl utility to check for data_array attributes in the incoming traces. Here, only data_array attributes would have to be truncated.

cpsievert · 2016-05-31T05:40:54Z

test/jasmine/assets/transforms/groupby.js

+        }
+
+        newTrace.name = groupName;
+        newTrace.marker.color = opts.groupColors[groupName];


We should probably set the color for each relevant trace mode, i.e.

for (var i = 0; i < modes.length; i++) { newTrace[modes[i]].color = opts.groupColors[groupName]; }

timelyportfolio · 2016-06-29T02:14:13Z

looks like this pull is still alive and kicking. In terms of transforms, I think the vega vocabulary is fairly complete.

vega transform code

I assume though that the supporting datalib is probably beyond the scope of plotly.js.

I wonder how we might combine group_by with filter, such that the groups are true and false based on the evaluation of the filter. I think this will be a common use case in R as we try to achieve inter-widget communication.

Also, one other thought is we might want to look at how dc.js handles its filters. As a good test, I would like to try to splice a dc.js chart with plotly.js.

- https://github.com/plotly/plotly.js/pull/491/files#diff-ad4f76ccd6044ed16514297078e13b84R455 for more detail on the conflict

mcin-armintaheri-archive · 2016-07-08T20:16:47Z

I gotta say, this transform stuff is working quite well!

Though there may be a bug, I'm still trying to figure it out:

I'm not 100% sure why this happens, but for any transform you make, if you add a new trace during the transform, you can't hide the trace by clicking it in the legend :s

I've only tried this by adding a new scatter trace with mode lines.

etpinard · 2016-07-08T20:24:30Z

@armintaheri

I'm not 100% sure why this happens, but for any transform you make, if you add a new trace during the transform, you can't hide the trace by clicking it in the legend

Yep, that's a known bug. See #499 (comment)

If you have any insight on what the desired behaviour should be, let me know.

timelyportfolio · 2016-07-08T20:24:51Z

@armintaheri, the legend seems to get fixed when I merged in @rreusser's animate-api-take-3.

etpinard · 2016-07-08T20:29:59Z

@timelyportfolio

the legend seems to get fixed when I merged in @rreusser's animate-api-take-3.

I doubt it. That would be some crazy magic fix from 🙏 if that's the case.

etpinard · 2016-07-19T15:19:26Z

Yeah @timelyportfolio the transforms included in this PR are meant for testing purposes only. Looking forward to seeing what you come up with 🍻

etpinard · 2016-07-19T15:59:02Z

@bpostlethwaite please review commits (off a different branch):

- remove error when array container are not of the same length - this is a temporary fix until #749 is addressed.

- move traceOut.index and traceOut._input assignments out of supplyTraceDefaults for clarity - 'index' ref -> index in user data (as before) - '_input' ref -> trace in user data (as before) - '_expandedIndex' ref -> index in fullData - '_expandedInput' ref -> full trace after transform, before 2nd trace dflts

- so that the 'same' expanded trace keep the same uid on update.

- module.transform must be a function - module.attributes and module.supplyDefaults are recommended but optional - log warning if transform module associated with transform type isn't found - extend traceIn if no supplyDefaults method is found.

timelyportfolio · 2016-07-20T17:28:58Z

@etpinard, should I be using transform-plugins or transform-plugins-2?

etpinard · 2016-07-20T18:47:05Z

@bpostlethwaite @chriddyp @timelyportfolio @rreusser @armintaheri this PR is now ready for review.

…ing" This reverts commit a06441a.

assets/ is ambiguous as now test/images/assets/ exists

bpostlethwaite · 2016-07-21T23:48:45Z

nice work! New age of hackerdackery for Plotly.js 💃 !

etpinard · 2016-08-04T14:04:18Z

@timelyportfolio have you made any progress on your transform modules?

timelyportfolio · 2016-08-04T14:43:51Z

@etpinard, yes, in multiple branches. Yesterday, I started new branch off master and added the filter with additional in \ notin and within \ notwithin (name suggestions welcome) and your groupby. @rreusser and I have been testing, playing, experimenting as we move along with his animate pull.

I moved transforms to src/transforms as you suggested, registered filter by default, and built to insure that all will work as we add other transforms.

On groupby I would like to work with you with the uuid to properly identify the sub-traces for things like the legend.

In terms of a fuller discussion in case we want to really push it, I just set up this Wiki https://github.com/timelyportfolio/plotly.js/wiki/Discussion---Transforms.

etpinard · 2016-08-04T14:46:39Z

@timelyportfolio thanks for the info!

etpinard added 10 commits May 3, 2016 12:24

expose Lib on test dashboard window

53a5971

lint

9909382

add check so that we don't try to destroy gd if it DNE

28147e2

make Plotly.register handle transform modules

9ac6d0f

make Plotly.restyle support transforms:

8ab9d0c

- the 'traces' array of trace index is now based off gd.data - make all transforms restyle calls go through doCalcdata which is the worse case scenario (i.e. we take no chance!)

add two example transform modules

2697975

add transform test suite

56665f2

update tests (plots.supplyDataDefaults -> supplyTraceDefaults)

aa274ef

[tmp commit] add example transforms to ./lib/ Plotly for testing

a06441a

etpinard added feature something new status: in progress labels May 3, 2016

alexcjohnson reviewed May 3, 2016
View reviewed changes

jcheng5 reviewed May 9, 2016
View reviewed changes

cpsievert mentioned this pull request May 10, 2016

Multiple mapping variables leads to overplotting plotly/plotly.R#577

Closed

etpinard mentioned this pull request May 25, 2016

Sampling (with LTTB or other algorithm) for performance on large data sets #560

Closed

cpsievert reviewed May 31, 2016
View reviewed changes

etpinard added 3 commits July 4, 2016 16:53

Merge branch 'master' into transform-plugins

0b62e45

- https://github.com/plotly/plotly.js/pull/491/files#diff-ad4f76ccd6044ed16514297078e13b84R455 for more detail on the conflict

toimage_test: make sure that all graph divs are deleted

1c651e8

plots: rm useless skip-over-step

f93b9c1

etpinard added 10 commits July 19, 2016 15:47

plots: make relinkPrivateKeys less strict

37e291b

- remove error when array container are not of the same length - this is a temporary fix until #749 is addressed.

move common code in pushModule func

383c6d4

rm useless comment about fits ;)

be36402

do not add transforms container if fullTrace is no transform present

0e1eb5a

mutate uid of expanded traces w.r.t to expanded index:

9a341c1

- so that the 'same' expanded trace keep the same uid on update.

make uid of expanded trace valid alpha-numerical hashes

1752d30

add support for transform in plot schema and Plotly.validate

b5ec01e

improve validate test cases

a6fda02

improve transform tests

8d7dc46

etpinard added status: reviewable and removed status: in progress labels Jul 20, 2016

etpinard added 2 commits July 20, 2016 14:49

Revert "[tmp commit] add example transforms to ./lib/ Plotly for test…

320d62e

…ing" This reverts commit a06441a.

rm @assets path shortcut,

3e6e9c1

assets/ is ambiguous as now test/images/assets/ exists

etpinard merged commit e020bb1 into master Jul 22, 2016

etpinard deleted the transform-plugins branch July 22, 2016 13:33

monfera mentioned this pull request Sep 8, 2016

Transform input data: groupby, filter #917

Closed

2 tasks

This was referenced Sep 29, 2016

Provide a data table view usable via JavaScript #991

Closed

Custom scales or more advanced hoverinfo support #1008

Closed

etpinard mentioned this pull request Nov 1, 2016

Plot from .json file #1104

Closed

tomwayson mentioned this pull request May 24, 2017

amCharts vs plot.ly Esri/cedar#266

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing transform plugins #499

Introducing transform plugins #499

etpinard commented May 3, 2016 •

edited

Loading

etpinard commented May 3, 2016 •

edited

Loading

alexcjohnson May 3, 2016

alexcjohnson May 3, 2016

etpinard May 3, 2016

chriddyp commented May 3, 2016

jcheng5 May 9, 2016

etpinard Jul 19, 2016

cpsievert May 31, 2016

timelyportfolio commented Jun 29, 2016 •

edited

Loading

mcin-armintaheri-archive commented Jul 8, 2016

etpinard commented Jul 8, 2016

timelyportfolio commented Jul 8, 2016

etpinard commented Jul 8, 2016

etpinard commented Jul 19, 2016

etpinard commented Jul 19, 2016 •

edited

Loading

timelyportfolio commented Jul 20, 2016

etpinard commented Jul 20, 2016

bpostlethwaite commented Jul 21, 2016

etpinard commented Aug 4, 2016

timelyportfolio commented Aug 4, 2016

etpinard commented Aug 4, 2016

Introducing transform plugins #499

Introducing transform plugins #499

Conversation

etpinard commented May 3, 2016 • edited Loading

In brief

Examples

Does it work?

TODOs

etpinard commented May 3, 2016 • edited Loading

EDIT

alexcjohnson May 3, 2016

Choose a reason for hiding this comment

alexcjohnson May 3, 2016

Choose a reason for hiding this comment

etpinard May 3, 2016

Choose a reason for hiding this comment

chriddyp commented May 3, 2016

jcheng5 May 9, 2016

Choose a reason for hiding this comment

etpinard Jul 19, 2016

Choose a reason for hiding this comment

cpsievert May 31, 2016

Choose a reason for hiding this comment

timelyportfolio commented Jun 29, 2016 • edited Loading

mcin-armintaheri-archive commented Jul 8, 2016

etpinard commented Jul 8, 2016

timelyportfolio commented Jul 8, 2016

etpinard commented Jul 8, 2016

etpinard commented Jul 19, 2016

etpinard commented Jul 19, 2016 • edited Loading

timelyportfolio commented Jul 20, 2016

etpinard commented Jul 20, 2016

bpostlethwaite commented Jul 21, 2016

etpinard commented Aug 4, 2016

timelyportfolio commented Aug 4, 2016

etpinard commented Aug 4, 2016

etpinard commented May 3, 2016 •

edited

Loading

etpinard commented May 3, 2016 •

edited

Loading

timelyportfolio commented Jun 29, 2016 •

edited

Loading

etpinard commented Jul 19, 2016 •

edited

Loading