Merge pull request #2737 from plotly/full_fig

full_figure_for_development

Author: nicolaskruchten

Diff for: .circleci/config.yml

@@ -9,7 +9,7 @@ jobs:
       - checkout
       - run:
           name: Install black
-          command: "sudo pip install black"
+          command: "sudo pip install black==19.10b0"
       - run:
           name: Check formatting with black
           command: "black --check ."

Diff for: CHANGELOG.md

@@ -4,8 +4,9 @@ This project adheres to [Semantic Versioning](http://semver.org/).
 
 ## [4.10.0] - unreleased
 
+### Added
 
-## [4.9.1] - unreleased
+- Added `plotly.io.full_figure_for_development()` and `plotly.graph_objects.Figure.full_figure_for_development()` ([#2737](https://github.com/plotly/plotly.py/pull/2737))
 
 
 ## [4.9.0] - 2020-07-16

Diff for: doc/python/figure-introspection.md

@@ -0,0 +1,175 @@
+---
+jupyter:
+  jupytext:
+    notebook_metadata_filter: all
+    text_representation:
+      extension: .md
+      format_name: markdown
+      format_version: '1.2'
+      jupytext_version: 1.4.2
+  kernelspec:
+    display_name: Python 3
+    language: python
+    name: python3
+  language_info:
+    codemirror_mode:
+      name: ipython
+      version: 3
+    file_extension: .py
+    mimetype: text/x-python
+    name: python
+    nbconvert_exporter: python
+    pygments_lexer: ipython3
+    version: 3.7.7
+  plotly:
+    description: How to dig into and learn more about the figure data structure.
+    display_as: file_settings
+    language: python
+    layout: base
+    name: Introspecting Figures
+    order: 35
+    page_type: u-guide
+    permalink: python/figure-introspection/
+    thumbnail: thumbnail/violin.jpg
+---
+
+### The Figure Lifecycle
+
+As explained in the [Figure Data Structure documentation](/python/figure-structure/), when building a figure object with Plotly.py, it is not necessary to populate every possible attribute. At render-time, figure objects (whether generated via [Plotly Express](/python/plotly-express/) or [Graph Objects](/python/graph-objects/) are passed from Plotly.py to [Plotly.js](/javascript/), which is the Javascript library responsible for turning JSON descriptions of figures into graphical representations. 
+
+As part of this rendering process, Plotly.js will determine, based on the attributes that have been set, which other attributes require values in order to draw the figure. Plotly.js will then apply either static or dynamic defaults to all of the remaining required attributes and render the figure. A good example of a static default would be the text font size: if unspecified, the default value is always the same. A good example of a dynamic default would be the range of an axis: if unspecified, the default will be computed based on the range of the data in traces associated with that axis.
+
+
+### Introspecting Plotly Express Figures
+
+Figure objects created by [Plotly Express](/python/plotly-express/) have a number of attributes automatically set, and these can be introspected using the Python `print()` function, or in JupyterLab, the special `fig.show("json")` renderer, which gives an interactive drilldown interface with search:
+
+```python
+import plotly.express as px
+
+fig = px.scatter(x=[10, 20], y=[20, 10], height=400, width=400)
+fig.show()
+print(fig)
+```
+
+We can learn more about the attributes Plotly Express has set for us with the Python `help()` function:
+
+```python
+help(fig.data[0].__class__.mode)
+```
+
+### Accessing Javascript-Computed Defaults
+
+_new in 4.10_
+
+The `.full_figure_for_development()` method provides Python-level access to the default values computed by Plotly.js. This method requires [the Kaleido package](/python/static-image-export/), which is easy to install and also used for [static image export](/python/static-image-export/).
+
+By way of example, here is an extremely simple figure created with [Graph Objects](/python/graph-objects/) (although it could have been made with [Plotly Express](/python/plotly-express/) as well just like above) where we have disabled the default template for maximum readability. Note how in this figure the text labels on the markers are clipped, and sit on top of the markers.
+
+```python
+import plotly.graph_objects as go
+
+fig = go.Figure(
+    data=[go.Scatter(
+        mode="markers+text", 
+        x=[10,20], 
+        y=[20, 10], 
+        text=["Point A", "Point B"]
+    )],
+    layout=dict(height=400, width=400, template="none")
+)
+fig.show()
+```
+
+Let's print this figure to see the very small JSON object that is passed to Plotly.js as input:
+
+```python
+print(fig)
+```
+
+Now let's look at the "full" figure after Plotly.js has computed the default values for every necessary attribute.
+
+> Heads-up: the full figure is quite long and intimidating, and this page is meant to help demystify things so **please read on**! 
+
+Please also note that the `.full_figure_for_development()` function is really meant for interactive learning and debugging, rather than production use, hence its name and the warning it produces by default, which you can see below, and which can be supressed with `warn=False`.
+
+```python
+full_fig = fig.full_figure_for_development()
+print(full_fig)
+```
+
+As you can see, Plotly.js does a lot of work filling things in for us! Let's look at the examples described at the top of the page of static and dynamic defaults. If we look just at `layout.font` and `layout.xaxis.range` we can see that the static default font size is 12 and that the dynamic default range is computed to be a bit beyond the data range which was 10-20:
+
+```python
+print("full_fig.layout.font.size: ", full_fig.layout.font.size)
+print("full_fig.layout.xaxis.range: ", full_fig.layout.xaxis.range)
+```
+
+### Learning About Attributes
+
+
+What else can we use this `full_fig` for? Let's start by looking at the first entry of the `data` 
+
+```python
+print(full_fig.data[0])
+```
+
+We see that this is an instance of `go.Scatter` (as expected, given the input) and that it has an attribute we've maybe never heard of called `cliponaxis` which by default seems to be set to `True` in this case. Let's find out more about this attribute using the built-in Python `help()` function
+
+```python
+help(go.Scatter.cliponaxis)
+```
+
+Aha!  This explains why in our original figure above, the text was cut off by the edge of the plotting area! Let's try forcing that to `False`, and let's also use the attribute `textposition` which we see in the full figure is by default set to `"middle center"` to get our text off of our markers:
+
+```python
+fig.update_traces(cliponaxis=False, textposition="top right")
+fig.show()
+```
+
+We can use this technique (of making a figure, and querying Plotly.js for the "full" version of that figure, and then exploring the attributes that are automatically set for us) to learn more about the range of possibilities that the figure schema makes available. We can drill down into `layout` attributes also:
+
+```python
+help(go.layout.XAxis.autorange)
+```
+
+### More about Layout
+
+In the figure we introspected above, we had added [a `scatter` trace](/python/line-and-scatter/), and Plotly.js automatically filled in for us the `xaxis` and `yaxis` values of that trace object to be `x` and `y`, and then also filled out the corresponding `layout.xaxis` and `layout.yaxis` objects for us, complete with their [extensive set of defaults for gridlines, tick labels and so on](/python/axes/).
+
+If we create a figure with [a `scattergeo` trace](/python/scatter-plots-on-maps/) instead, however, Plotly.js will fill in a totally different set of objects in `layout`, corresponding to [a `geo` subplot, with all of its defaults for whether or not to show rivers, lakes, country borders, coastlines etc](https://plotly.com/python/map-configuration/).
+
+```python
+import plotly.graph_objects as go
+
+fig = go.Figure(
+    data=[go.Scattergeo(
+        mode="markers+text", 
+        lat=[10, 20], 
+        lon=[20, 10], 
+        text=["Point A", "Point B"]
+    )],
+    layout=dict(height=400, width=400, 
+                margin=dict(l=0,r=0,b=0,t=0),
+                template="none")
+)
+fig.show()
+full_fig = fig.full_figure_for_development()
+print(full_fig)
+```
+
+If I then set `showrivers=True` and re-query the full figure, I see that new keys have appeared in the `layout.geo` object for `rivercolor` and `riverwidth`, showing the dynamic nature of these defaults.
+
+```python
+fig.update_geos(showrivers=True)
+full_fig = fig.full_figure_for_development()
+print(full_fig.layout.geo)
+```
+
+### Reference
+
+You can learn more about [all the available attributes in the plotly figure schema](/python/reference/) (and read about its [high-level structure](/python/figure-structure/)) or about [all the classes and functions in the `plotly` module](/python-api-reference/).
+
+```python
+
+```

Diff for: doc/python/figure-structure.md

@@ -57,7 +57,7 @@ Attributes are referred to in text and in the [Figure Reference](/python/referen
 
 The [`plotly.graph_objects` module contains an automatically-generated hierarchy of  Python classes](/python/graph-objects/) which represent non-leaf attributes in the figure schema and provide a Pythonic API for them. When [manipulating a `plotly.graph_objects.Figure` object](/python/creating-and-updating-figures/), attributes can be set either directly using Python object attributes e.g. `fig.layout.title.font.family="Open Sans"` or using [update methods and "magic underscores"](/python/creating-and-updating-figures/#magic-underscore-notation) e.g. `fig.update_layout(title_font_family="Open Sans")`
 
-When building a figure, it is *not necessary to populate every attribute* of every object. At render-time, the JavaScript layer will compute default values for each required unspecified attribute, depending upon the ones that are specified, as documented in the [Figure Reference](/python/reference/). An example of this would be `layout.xaxis.range`, which may be specified explicitly, but if not will be computed based on the range of `x` values for every trace linked to that axis. The JavaScript layer will ignore unknown attributes or malformed values, although the `plotly.graph_objects` module provides Python-side validation for attribute values. Note also that if [the `layout.template` key is present (as it is by default)](/python/templates/) then default values will be drawn first from the contents of the template and only if missing from there will the JavaScript layer infer further defaults. The built-in template can be disabled by setting `layout.template="none"`.
+When building a figure, it is *not necessary to populate every attribute* of every object. At render-time, [the JavaScript layer will compute default values](/python/figure-introspection/) for each required unspecified attribute, depending upon the ones that are specified, as documented in the [Figure Reference](/python/reference/). An example of this would be `layout.xaxis.range`, which may be specified explicitly, but if not will be computed based on the range of `x` values for every trace linked to that axis. The JavaScript layer will ignore unknown attributes or malformed values, although the `plotly.graph_objects` module provides Python-side validation for attribute values. Note also that if [the `layout.template` key is present (as it is by default)](/python/templates/) then default values will be drawn first from the contents of the template and only if missing from there will the JavaScript layer infer further defaults. The built-in template can be disabled by setting `layout.template="none"`.
 
 ### The Top-Level `data` Attribute
 

Diff for: doc/requirements.txt

@@ -25,3 +25,4 @@ python-frontmatter
 datashader
 pyarrow
 cufflinks==0.17.3
+kaleido

Diff for: packages/python/plotly/plotly/basedatatypes.py

@@ -1104,7 +1104,7 @@ def _add_annotation_like(
             if refs[0].subplot_type != "xy":
                 raise ValueError(
                     """
-Cannot add {prop_singular} to subplot at position ({r}, {c}) because subplot 
+Cannot add {prop_singular} to subplot at position ({r}, {c}) because subplot
 is of type {subplot_type}.""".format(
                         prop_singular=prop_singular,
                         r=row,
@@ -2896,6 +2896,35 @@ def to_json(self, *args, **kwargs):
 
         return pio.to_json(self, *args, **kwargs)
 
+    def full_figure_for_development(self, warn=True, as_dict=False):
+        """
+        Compute default values for all attributes not specified in the input figure and
+        returns the output as a "full" figure. This function calls Plotly.js via Kaleido
+        to populate unspecified attributes. This function is intended for interactive use
+        during development to learn more about how Plotly.js computes default values and is
+        not generally necessary or recommended for production use.
+
+        Parameters
+        ----------
+        fig:
+            Figure object or dict representing a figure
+
+        warn: bool
+            If False, suppress warnings about not using this in production.
+
+        as_dict: bool
+            If True, output is a dict with some keys that go.Figure can't parse.
+            If False, output is a go.Figure with unparseable keys skipped.
+
+        Returns
+        -------
+        plotly.graph_objects.Figure or dict
+            The full figure
+        """
+        import plotly.io as pio
+
+        return pio.full_figure_for_development(self, warn, as_dict)
+
     def write_json(self, *args, **kwargs):
         """
         Convert a figure to JSON and write it to a file or writeable

Diff for: packages/python/plotly/plotly/io/__init__.py

@@ -2,7 +2,7 @@
 import sys
 
 if sys.version_info < (3, 7):
-    from ._kaleido import to_image, write_image
+    from ._kaleido import to_image, write_image, full_figure_for_development
     from . import orca, kaleido
     from ._json import to_json, from_json, read_json, write_json
     from ._templates import templates, to_templated
@@ -25,6 +25,7 @@
         "renderers",
         "show",
         "base_renderers",
+        "full_figure_for_development",
     ]
 else:
     __all__, __getattr__, __dir__ = relative_import(
@@ -33,6 +34,7 @@
         [
             "._kaleido.to_image",
             "._kaleido.write_image",
+            "._kaleido.full_figure_for_development",
             "._json.to_json",
             "._json.from_json",
             "._json.read_json",

Diff for: packages/python/plotly/plotly/io/_kaleido.py

@@ -1,6 +1,7 @@
 from __future__ import absolute_import
 from six import string_types
 import os
+import json
 import plotly
 from plotly.io._utils import validate_coerce_fig_to_dict
 
@@ -120,7 +121,7 @@ def to_image(
             """
 Image export using the "kaleido" engine requires the kaleido package,
 which can be installed using pip:
-    $ pip install -U kaleido 
+    $ pip install -U kaleido
 """
         )
 
@@ -260,4 +261,58 @@ def write_image(
         file.write(img_data)
 
 
-__all__ = ["to_image", "write_image", "scope"]
+def full_figure_for_development(fig, warn=True, as_dict=False):
+    """
+    Compute default values for all attributes not specified in the input figure and
+    returns the output as a "full" figure. This function calls Plotly.js via Kaleido
+    to populate unspecified attributes. This function is intended for interactive use
+    during development to learn more about how Plotly.js computes default values and is
+    not generally necessary or recommended for production use.
+
+    Parameters
+    ----------
+    fig:
+        Figure object or dict representing a figure
+
+    warn: bool
+        If False, suppress warnings about not using this in production.
+
+    as_dict: bool
+        If True, output is a dict with some keys that go.Figure can't parse.
+        If False, output is a go.Figure with unparseable keys skipped.
+
+    Returns
+    -------
+    plotly.graph_objects.Figure or dict
+        The full figure
+    """
+
+    # Raise informative error message if Kaleido is not installed
+    if scope is None:
+        raise ValueError(
+            """
+Full figure generation requires the kaleido package,
+which can be installed using pip:
+    $ pip install -U kaleido
+"""
+        )
+
+    if warn:
+        import warnings
+
+        warnings.warn(
+            "full_figure_for_development is not recommended or necessary for "
+            "production use in most circumstances. \n"
+            "To suppress this warning, set warn=False"
+        )
+
+    fig = json.loads(scope.transform(fig, format="json").decode("utf-8"))
+    if as_dict:
+        return fig
+    else:
+        import plotly.graph_objects as go
+
+        return go.Figure(fig, skip_invalid=True)
+
+
+__all__ = ["to_image", "write_image", "scope", "full_figure_for_development"]

Diff for: packages/python/plotly/plotly/tests/test_optional/test_kaleido/test_kaleido.py

@@ -28,6 +28,12 @@ def test_kaleido_engine_to_image_returns_bytes():
     assert result.startswith(b"<svg")
 
 
+def test_kaleido_fulljson():
+    empty_fig = dict(data=[], layout={})
+    result = pio.full_figure_for_development(empty_fig, warn=False, as_dict=True)
+    assert result["layout"]["calendar"] == "gregorian"
+
+
 def test_kaleido_engine_to_image():
     with mocked_scope() as scope:
         pio.to_image(fig, engine="kaleido", validate=False)

Diff for: packages/python/plotly/plotly/tests/test_orca/images/darwin/fig1.eps

@@ -1,5 +1,5 @@
 %!PS-Adobe-3.0 EPSF-3.0
-%Produced by poppler pdftops version: 0.65.0 (http://poppler.freedesktop.org)
+%Produced by poppler pdftops version: 0.81.0 (http://poppler.freedesktop.org)
 %%Creator: Chromium
 %%LanguageLevel: 2
 %%DocumentSuppliedResources: (atend)

Diff for: packages/python/plotly/plotly/tests/test_orca/images/darwin/latexfig.eps

@@ -1,5 +1,5 @@
 %!PS-Adobe-3.0 EPSF-3.0
-%Produced by poppler pdftops version: 0.65.0 (http://poppler.freedesktop.org)
+%Produced by poppler pdftops version: 0.81.0 (http://poppler.freedesktop.org)
 %%Creator: Chromium
 %%LanguageLevel: 2
 %%DocumentSuppliedResources: (atend)

Diff for: packages/python/plotly/plotly/tests/test_orca/images/darwin/topofig.eps

@@ -1,5 +1,5 @@
 %!PS-Adobe-3.0 EPSF-3.0
-%Produced by poppler pdftops version: 0.65.0 (http://poppler.freedesktop.org)
+%Produced by poppler pdftops version: 0.81.0 (http://poppler.freedesktop.org)
 %%Creator: Chromium
 %%LanguageLevel: 2
 %%DocumentSuppliedResources: (atend)