Skip to content

Animations/Frames in Python API #584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Nov 26, 2016
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
cb4552c
iplot in offline works with frames
Kully Oct 18, 2016
b4c3953
updated js version in plotly offline
Kully Oct 19, 2016
54c1386
added Andrew's #586 PR
Kully Oct 20, 2016
d19c6ca
added some print statements to check progresses
Kully Oct 21, 2016
efa390e
__init__.py now accepts create_animations which lives in plotly.py --…
Kully Oct 28, 2016
eb57218
rewrote create_animations to take frames - still beta
Kully Nov 3, 2016
5482b53
merged master offline.py to local - changed call signature in one func
Kully Nov 3, 2016
b2b8451
tests
Kully Nov 3, 2016
a156503
added create_animations so v2 is hit if frames is in fig
Kully Nov 4, 2016
d173cbb
create_animations to bad_create_animations
Kully Nov 8, 2016
d17a222
added function to spit out uids from column names
Kully Nov 14, 2016
49bdc97
made get_grid() to pull grid from cloud and added method in Grid clas…
Kully Nov 15, 2016
4fbdc99
cleaned up url parsing + moved json_to_grid functionality to __init__…
Kully Nov 16, 2016
dbb386f
fixed order upon creating Grid
Kully Nov 17, 2016
bd2132b
accounted for orders with jump discontinuities
Kully Nov 18, 2016
26c3714
moved all_columns list where appropriate - above code block that uses it
Kully Nov 19, 2016
0c7bb1c
...
Kully Nov 24, 2016
4acc146
commented out fid stuff
Kully Nov 24, 2016
c195e78
merged master into branch
Kully Nov 24, 2016
6f12ba8
delete print lines from debugging
Kully Nov 24, 2016
253228f
grid_id is set to grids created via get_grid
Kully Nov 24, 2016
405e71d
add 'Frames' to grid_objs_test
Kully Nov 24, 2016
c1727d8
removed commented out code for old get_uid code
Kully Nov 24, 2016
fca8fed
added animations function
Kully Nov 24, 2016
587dfa5
Chris' comments sans tests
Kully Nov 25, 2016
495b015
changed some terminology around
Kully Nov 26, 2016
dd20177
last of non-blocking
Kully Nov 26, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions plotly/graph_objs/graph_objs.py
Original file line number Diff line number Diff line change
Expand Up @@ -836,6 +836,20 @@ def __init__(self, *args, **kwargs):
_parent_key='data')
figure_class.__init__ = __init__

def __setitem__(self, key, value, **kwargs):
if key == 'frames':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a comment about this one. Maybe tie it to a new issue to better-integrate frames into Figure? As always, pls put the issue numer (or link) in the TODO comment.

super(PlotlyDict, self).__setitem__(key, value)
else:
super(figure_class, self).__setitem__(key, value, **kwargs)
figure_class.__setitem__ = __setitem__

def _get_valid_attributes(self):
super(figure_class, self)._get_valid_attributes()
if 'frames' not in self._valid_attributes:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto hear, link it to that same issue ^^

self._valid_attributes.add('frames')
return self._valid_attributes
figure_class._get_valid_attributes = _get_valid_attributes

def get_data(self, flatten=False):
"""
Returns the JSON for the plot with non-data elements stripped.
Expand Down
4 changes: 1 addition & 3 deletions plotly/graph_reference.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
'ErrorZ': {'object_name': 'error_z', 'base_type': dict},
'Figure': {'object_name': 'figure', 'base_type': dict},
'Font': {'object_name': 'font', 'base_type': dict},
'Frames': {'object_name': 'frames', 'base_type': dict},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? Did I accidentally leave this in or something ;P?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I added it in. I don't know if it's actually being used though.

'Heatmap': {'object_name': 'heatmap', 'base_type': dict},
'Histogram': {'object_name': 'histogram', 'base_type': dict},
'Histogram2d': {'object_name': 'histogram2d', 'base_type': dict},
Expand Down Expand Up @@ -87,7 +88,6 @@ def get_graph_reference():

graph_reference_url = '{}{}?sha1={}'.format(plotly_api_domain,
GRAPH_REFERENCE_PATH, sha1)

try:
response = requests.get(graph_reference_url,
timeout=GRAPH_REFERENCE_DOWNLOAD_TIMEOUT)
Expand Down Expand Up @@ -196,11 +196,9 @@ def get_valid_attributes(object_name, parent_object_names=()):
# These are for documentation and quick lookups. They're just strings.
valid_attributes = set()
for attributes_dict in attributes.values():

for key, val in attributes_dict.items():
if key not in GRAPH_REFERENCE['defs']['metaKeys']:
valid_attributes.add(key)

deprecated_attributes = attributes_dict.get('_deprecated', {})
for key, val in deprecated_attributes.items():
if key not in GRAPH_REFERENCE['defs']['metaKeys']:
Expand Down
51 changes: 40 additions & 11 deletions plotly/offline/offline.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,8 +162,15 @@ def init_notebook_mode(connected=False):

def _plot_html(figure_or_data, show_link, link_text, validate,
default_width, default_height, global_requirejs):

figure = tools.return_figure_from_figure_or_data(figure_or_data, validate)
# force no validation if frames is in the call
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO this and link to some issue pls :)

if 'frames' in figure_or_data:
figure = tools.return_figure_from_figure_or_data(
figure_or_data, False
)
else:
figure = tools.return_figure_from_figure_or_data(
figure_or_data, validate
)

width = figure.get('layout', {}).get('width', default_width)
height = figure.get('layout', {}).get('height', default_height)
Expand All @@ -185,6 +192,8 @@ def _plot_html(figure_or_data, show_link, link_text, validate,
plotdivid = uuid.uuid4()
jdata = json.dumps(figure.get('data', []), cls=utils.PlotlyJSONEncoder)
jlayout = json.dumps(figure.get('layout', {}), cls=utils.PlotlyJSONEncoder)
if 'frames' in figure_or_data:
jframes = json.dumps(figure.get('frames', {}), cls=utils.PlotlyJSONEncoder)

config = {}
config['showLink'] = show_link
Expand All @@ -204,11 +213,30 @@ def _plot_html(figure_or_data, show_link, link_text, validate,
link_text = link_text.replace('plot.ly', link_domain)
config['linkText'] = link_text

script = 'Plotly.newPlot("{id}", {data}, {layout}, {config})'.format(
id=plotdivid,
data=jdata,
layout=jlayout,
config=jconfig)
if 'frames' in figure_or_data:
script = '''
Plotly.plot(
'{id}',
{data},
{layout},
{config}
).then(function () {add_frames}).then(function(){animate})
Copy link

@rreusser rreusser Oct 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See: plotly/plotly.js#1054

That PR has been merged and is on plotly.js/master (unreleased, but coming very soon?) and makes the syntax somewhat simpler. I'd elaborate here, but the PR description explains it fairly clearly. Calling addFrames separately will remain perfectly fine, but ideally will become unnecessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, fantastic. I think Chris mentioned that I'd have to do some rewriting when that gets added.

I'll leave it for now, but does that mean frames is available as a "valid attribute" for Python?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, should just be a simplification. Long story short, given data, layout, frames:

  • frames[i].data follows the same validation as data itself
  • frames[i].layout follows the same validation as layout

I'm not 100% sure what that means for validation, but on a strictly abstract level, it's conceptually simple.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry, unless you were referring to python arguments instead of data validation… That PR just changes the call signature. Not 100% sure the other concerns, but feel free to ask whatever questions you need and I'll do my best to get you what you need.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm referring to this py.iplot(fig) call where fig contains data, layout and frames. Right now, I'm still getting that:

'frames' is not allowed in 'figure'

Path To Error: ['frames']

Valid attributes for 'figure' at path [] under parents []:

    ['layout', 'data']

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, got it. Yeah, it should permit frames.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still receiving the error. Anything I can do to fix? I sudo pip upgraded just now and it reported no new changes.

Copy link

@rreusser rreusser Oct 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm… not sure. It seems like that's a whitelisting issue on the plotly (not js) side.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, so after playing follow-that-code for a while, I've traced it to the v2 plot schema Can we add frames to this?

'''.format(
id=plotdivid,
data=jdata,
layout=jlayout,
config=jconfig,
add_frames="{" + "return Plotly.addFrames('{id}',{frames}".format(
id=plotdivid, frames=jframes
) + ");}",
animate="{" + "Plotly.animate('{id}');".format(id=plotdivid) + "}"
)
else:
script = 'Plotly.newPlot("{id}", {data}, {layout}, {config})'.format(
id=plotdivid,
data=jdata,
layout=jlayout,
config=jconfig)

optional_line1 = ('require(["plotly"], function(Plotly) {{ '
if global_requirejs else '')
Expand All @@ -232,6 +260,7 @@ def _plot_html(figure_or_data, show_link, link_text, validate,

return plotly_html_div, plotdivid, width, height


def iplot(figure_or_data, show_link=True, link_text='Export to plot.ly',
validate=True, image=None, filename='plot_image', image_width=800,
image_height=600):
Expand Down Expand Up @@ -301,10 +330,10 @@ def iplot(figure_or_data, show_link=True, link_text='Export to plot.ly',
)
# if image is given, and is a valid format, we will download the image
script = get_image_download_script('iplot').format(format=image,
width=image_width,
height=image_height,
filename=filename,
plot_id=plotdivid)
width=image_width,
height=image_height,
filename=filename,
plot_id=plotdivid)
# allow time for the plot to draw
time.sleep(1)
# inject code to download an image of the plot
Expand Down
112 changes: 58 additions & 54 deletions plotly/offline/plotly.min.js

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion plotly/plotly/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,6 @@
grid_ops,
meta_ops,
file_ops,
get_config
get_config,
create_animations
)
81 changes: 76 additions & 5 deletions plotly/plotly/plotly.py
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,6 @@ def plot(figure_or_data, validate=True, **plot_options):

"""
figure = tools.return_figure_from_figure_or_data(figure_or_data, validate)

for entry in figure['data']:
if ('type' in entry) and (entry['type'] == 'scattergl'):
continue
Expand Down Expand Up @@ -1399,6 +1398,7 @@ def _send_to_plotly(figure, **plot_options):
fig = tools._replace_newline(figure) # does not mutate figure
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚡️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been an empty comment for apparently one year. Should I attempt writing something in it and get it corrected?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oop, sorry. Weird. I just saw the change and questioned it. sure 🔪 it :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Write one or 'cut it'?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cut it is fine :)

data = json.dumps(fig['data'] if 'data' in fig else [],
cls=utils.PlotlyJSONEncoder)

credentials = get_credentials()
validate_credentials(credentials)
username = credentials['username']
Expand All @@ -1407,8 +1407,8 @@ def _send_to_plotly(figure, **plot_options):
fileopt=plot_options['fileopt'],
world_readable=plot_options['world_readable'],
sharing=plot_options['sharing'],
layout=fig['layout'] if 'layout' in fig
else {}),
layout=fig['layout'] if 'layout' in fig else {},
frames=fig['frames'] if 'frames' in fig else {}),
cls=utils.PlotlyJSONEncoder)

# TODO: It'd be cool to expose the platform for RaspPi and others
Expand All @@ -1420,12 +1420,13 @@ def _send_to_plotly(figure, **plot_options):
origin='plot',
kwargs=kwargs)

if 'frames' in kwargs:
r = create_animations(fig, kwargs, payload)

url = get_config()['plotly_domain'] + "/clientresp"

r = requests.post(url, data=payload,
verify=get_config()['plotly_ssl_verification'])
r.raise_for_status()
r = json.loads(r.text)

if 'error' in r and r['error'] != '':
raise exceptions.PlotlyError(r['error'])
Expand All @@ -1447,6 +1448,76 @@ def _send_to_plotly(figure, **plot_options):
return r


def create_animations(fig, kwargs, payload):
Copy link
Contributor Author

@Kully Kully Oct 28, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@theengineear @rreusser

Would appreciate some guidance...

I wrote something that can pretty reliably break apart a simple figure (2D Scatterplots for example) and make a grid then plot, and then a request to the V2 REST API. I't's working and I can view my stuff on the cloud.

I am getting an error when trying to pass frames into the figure for the plot generation: I just get some gargled HTML. What is the next step for me getting frames to pass?

A few more things I need to tackle:
-make sure I include things like sharing in the create_animations call.
-eventually, once frames is being allowed to exist in DATA, validate items in frames the same way I am here

Any thoughts/ideas are appreciated.

Copy link
Contributor

@theengineear theengineear Nov 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚨 💀 😨 I think you should not do figure splitting (i.e., exchanging raw data for referenced data). Please take my word on it. It's not worth it. It's extremely complicated (Please take my word on it).

^^ Seriously, take my word on it.

Users should be able to do something like:

grid = Grid()
grid.add_column('time_data', [1, 2, 3, 4])  # something like this...
grid.add_column('voltage', [120, 119, 122, 121])  # something like this...
# somehow, someway, add data to this grid and make it easy for users to pull out the reference.
# try looking in https://github.com/plotly/plotly.py/blob/master/plotly/grid_objs/grid_objs.py

figure.data[0].x = grid.time_data
figure.frames[0]['data'][0] = grid.time_data  # Not sure if this is right, just for example

create_animations(figure, ...)

AFAIK, you don't need to worry too much about checking for raw data in the figure. The api will catch you and the request will just fail.

Let's fully shift the responsibility to the user. If we get into the game of guessing, we'll just end up guessing wrong and creating bugs.

"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should tell the user that this is a beta endpoint that is subject to deprecation in the future.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And we should describe how this is similar and dissimilar to plotly.plotly.plot. Includes things like:

  • Folders aren't supported
  • Overwrite isn't supported
  • Animations via frames are supported
  • There might be some other things that I'm forgetting about.

The plan is to create a grid then a plot by deconstructing the fig if
frames is found in the kwargs.
"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should include some basic examples of animations in here.

url_v2_plot = "https://api.plot.ly/v2/plots"
url_v2_grid = "https://api.plot.ly/v2/grids"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💀 you need to get these urls from the config object. There should be examples of that somewhere. Here's one: https://github.com/plotly/plotly.py/blob/master/plotly/plotly/plotly.py#L685 (something like that should do).

In general, never hard-code the plot.ly domain into this code.

auth = HTTPBasicAuth(str(payload['un']), str(payload['key']))
headers = {'Plotly-Client-Platform': 'python'}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


# add layout if not in fig
if 'layout' not in fig:
fig['layout'] = {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary comment and this can be a clean one-liner as you've done before.


# make grid
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This, on the other hand, is a nice little comment, imo.

cols_dict = {}
counter = 0
trace_num = 0
for trace in fig['data']:
for var in ['x', 'y']:
if 'name' in trace:
cols_dict["{name}, {x_or_y}".format(name=trace['name'],
x_or_y=var)] = {
"data": list(trace[var]), "order": counter
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐄 if it's going to take up 4 lines anyhow, I'd suggest something like?

key = '{}, {}'.format(trace['name'], var)
data = list(trace[var])
cols[key] = {'data': data, 'order': counter}

else:
cols_dict["Trace {num}, {x_or_y}".format(num=trace_num,
x_or_y=var)] = {
"data": list(trace[var]), "order": counter
}
counter += 1
trace_num += 1

grid_info = {
"data": {"cols": cols_dict},
"world_readable": True
}

r = requests.post('https://api.plot.ly/v2/grids', auth=auth,
headers=headers, json=grid_info)
r_dict = json.loads(r.text)
# make a copy of the fig so that we not altering the original fig
fig_copy = copy.deepcopy(fig)

# make plot
fid = r_dict['file']['fid']
cols_index = 0
for trace in fig_copy['data']:
if 'x' in trace:
del trace['x']
if 'y' in trace:
del trace['y']

trace["xsrc"] = "{fid}:{idlocal}".format(
fid=fid, idlocal=r_dict['file']['cols'][cols_index]['uid']
)
trace["ysrc"] = "{fid}:{idlocal}".format(
fid=fid, idlocal=r_dict['file']['cols'][cols_index + 1]['uid']
)
cols_index += 2

plots_info = {
"figure": fig_copy,
"world_readable": True
}

return requests.post('https://api.plot.ly/v2/plots', auth=auth,
headers=headers, json=plots_info)


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add tests for this function including the possible types of errors that can occur

def _open_url(url):
try:
from webbrowser import open as wbopen
Expand Down