Merge branch 'master' into imshow-xarray

Author: emmanuelle

.circleci/config.yml

@@ -425,8 +425,8 @@ jobs:
             . venv/bin/activate
             echo ${mapbox_token} > python/.mapbox_token
             make -kj8 || make -kj8
-            curl https://raw.githubusercontent.com/plotly/documentation/source-design-merge/front-matter-ci.py > front-matter-ci.py
-            curl https://raw.githubusercontent.com/plotly/documentation/source-design-merge/check-or-enforce-order.py > check-or-enforce-order.py
+            curl https://raw.githubusercontent.com/plotly/graphing-library-docs/master/front-matter-ci.py > front-matter-ci.py
+            curl https://raw.githubusercontent.com/plotly/graphing-library-docs/master/check-or-enforce-order.py > check-or-enforce-order.py
             python front-matter-ci.py build/html
             python check-or-enforce-order.py build/html
             if [ "${CIRCLE_BRANCH}" == "doc-prod" ]; then
@@ -448,14 +448,14 @@ jobs:
           name: trigger doc build
           command: |
             if [ "${CIRCLE_BRANCH}" == "doc-prod" ]; then
-              git clone --depth=1 --branch=source-design-merge https://github.com/plotly/documentation.git
-              cd documentation
+              git clone --depth=1 https://github.com/plotly/graphing-library-docs.git
+              cd graphing-library-docs
               git config user.name plotlydocbot
               git config user.email [email protected]
               git commit --allow-empty -m "deploying https://github.com/plotly/plotly.py/commit/${CIRCLE_SHA1}"
               git push
               cd ..
-              rm -rf documentation
+              rm -rf graphing-library-docs
             fi
 
       - run:

binder/requirements.txt

@@ -12,3 +12,5 @@ psutil
 requests
 networkx
 scikit-image
+datashader
+pyarrow

contributing.md

@@ -213,7 +213,7 @@ This is the release process for releasing `plotly.py` version `X.Y.Z` with
 `plotlywidget` version `A.B.C`.
 
 Note: The `plotlywidget` instructions must be followed if any change
-has been made in the `js/` directory source code, OR if the version of
+has been made in the `packages/javascript` directory source code, OR if the version of
 plotly.js has been updated.  If neither of these is the case, there's no need
 to increment the `plotlywidget` version or to publish a new version to npm.
 
@@ -253,7 +253,9 @@ Note that the conda installation instructions must include
 "-c plotly/lable/test" rather than "-c plotly" in order to install the
 release candidate version.
 
-Commit Changelog and README updates.
+Update the `doc/python/getting-started.md` file with the same version numbers.
+
+Commit Changelog, README and getting-started updates.
 
 ### Bump to release candidate version
  1) Manually update the plotlywidget version to `A.B.C-rc.1` in the files
@@ -297,17 +299,19 @@ And, you'll need the credentials file `~/.pypirc`. Request access from
 (plotly_dev) $ twine upload dist/plotly-X.Y.Zrc1*
 ```
 
-### Publish release candidate of `plotlywidget` to NPM
+### Publish release candidate of `plotlywidget` and `jupyterlab-plotly` to NPM
 Now, publish the release candidate of the `plotlywidget` NPM package.
 
 ```bash
-cd ./js
+cd ./packages/javascript/plotlywidget
 npm publish --access public --tag next
 ```
 
 The `--tag next` part ensures that users won't install this version unless
 they explicitly ask for the version or for the version wtih the `next` tag.
 
+Do the same in the `jupyterlab-plotly` directory.
+
 ### Publish release candidate to plotly anaconda channel
 To publish package to the plotly anaconda channel you'll need to have the
 anaconda or miniconda distribution installed, and you'll need to have the
@@ -361,7 +365,9 @@ release candidate suffix from the following version strings:
 
  - `plotly/_widget_version.py`:
    + Update `__frontend_version__` to `^A.B.C` (Note the `^` prefix)
- - `js/package.json`
+ - `packages/javascript/plotlywidget/package.json`
+   + Update `"version"` to `A.B.C`
+ - `packages/javascript/jupyterlab-plotly/package.json`
    + Update `"version"` to `A.B.C`
 
 Commit and push to the release branch.
@@ -431,6 +437,19 @@ Make "Release title" the same string as the tag.
 
 Copy changelog section for this version as the "Describe this release"
 
+### Upgrade doc requirements and API doc
+
+Files to be updated:
+- `doc/apidoc/conf.py` with new version number
+- `doc/requirements.txt`
+- `binder/requirements.txt`
+
+### Synchronize master and doc-prod branches
+
+doc-prod should already have been merged on a regular basis into master, but
+start doing it first. Then merge master into doc-prod to deploy the doc related
+to features in the release.
+
 ### Post announcement
 Post a simple announcement to the Plotly Python forum, with links to the
 README installation instructions and to the CHANGELOG.

doc/python/3d-scatter-plots.md

@@ -35,7 +35,7 @@ jupyter:
 
 ## 3D scatter plot with Plotly Express
 
-[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/).
+[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/) and produces [easy-to-style figures](/python/styling-plotly-express/).
 
 Like the [2D scatter plot](https://plot.ly/python/line-and-scatter/) `px.scatter`, the 3D function `px.scatter_3d` plots individual data in three-dimensional space.
 

doc/python/bar-charts.md

@@ -5,7 +5,7 @@ jupyter:
     text_representation:
       extension: .md
       format_name: markdown
-      format_version: '1.2'
+      format_version: "1.2"
       jupytext_version: 1.3.0
   kernelspec:
     display_name: Python 3
@@ -35,7 +35,7 @@ jupyter:
 
 ### Bar chart with Plotly Express
 
-[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/).
+[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/) and produces [easy-to-style figures](/python/styling-plotly-express/).
 
 With `px.bar`, each row of the DataFrame is represented as a rectangular mark.
 

doc/python/box-plots.md

@@ -5,7 +5,7 @@ jupyter:
     text_representation:
       extension: .md
       format_name: markdown
-      format_version: '1.2'
+      format_version: "1.2"
       jupytext_version: 1.3.1
   kernelspec:
     display_name: Python 3
@@ -31,16 +31,16 @@ jupyter:
     page_type: example_index
     permalink: python/box-plots/
     redirect_from:
-    - /python/box/
-    - /python/basic_statistics/
+      - /python/box/
+      - /python/basic_statistics/
     thumbnail: thumbnail/box.jpg
 ---
 
 A [box plot](https://en.wikipedia.org/wiki/Box_plot) is a statistical representation of numerical data through their quartiles. The ends of the box represent the lower and upper quartiles, while the median (second quartile) is marked by a line inside the box. For other statistical representations of numerical data, see [other statistical charts](https://plot.ly/python/statistical-charts/).
 
 ## Box Plot with `plotly.express`
 
-[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/).
+[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/) and produces [easy-to-style figures](/python/styling-plotly-express/).
 
 In a box plot created by `px.box`, the distribution of the column given as `y` argument is represented.
 
@@ -73,13 +73,13 @@ fig.show()
 
 ### Choosing The Algorithm For Computing Quartiles
 
-By default, quartiles for box plots are computed using the `linear` method (for more about linear interpolation, see #10 listed on [http://www.amstat.org/publications/jse/v14n3/langford.html](http://www.amstat.org/publications/jse/v14n3/langford.html) and [https://en.wikipedia.org/wiki/Quartile](https://en.wikipedia.org/wiki/Quartile) for more details). 
+By default, quartiles for box plots are computed using the `linear` method (for more about linear interpolation, see #10 listed on [http://www.amstat.org/publications/jse/v14n3/langford.html](http://www.amstat.org/publications/jse/v14n3/langford.html) and [https://en.wikipedia.org/wiki/Quartile](https://en.wikipedia.org/wiki/Quartile) for more details).
 
-However, you can also choose to use an `exclusive` or an `inclusive` algorithm to compute quartiles. 
+However, you can also choose to use an `exclusive` or an `inclusive` algorithm to compute quartiles.
 
-The *exclusive* algorithm uses the median to divide the ordered dataset into two halves. If the sample is odd, it does not include the median in either half. Q1 is then the median of the lower half and Q3 is the median of the upper half.
+The _exclusive_ algorithm uses the median to divide the ordered dataset into two halves. If the sample is odd, it does not include the median in either half. Q1 is then the median of the lower half and Q3 is the median of the upper half.
 
-The *inclusive* algorithm also uses the median to divide the ordered dataset into two halves, but if the sample is odd, it includes the median in both halves. Q1 is then the median of the lower half and Q3 the median of the upper half.
+The _inclusive_ algorithm also uses the median to divide the ordered dataset into two halves, but if the sample is odd, it includes the median in both halves. Q1 is then the median of the lower half and Q3 the median of the upper half.
 
 ```python
 import plotly.express as px
@@ -92,7 +92,8 @@ fig.show()
 ```
 
 #### Difference Between Quartile Algorithms
-It can sometimes be difficult to see the difference between the linear, inclusive, and exclusive algorithms for computing quartiles. In the following example, the same dataset is visualized using each of the three different quartile computation algorithms. 
+
+It can sometimes be difficult to see the difference between the linear, inclusive, and exclusive algorithms for computing quartiles. In the following example, the same dataset is visualized using each of the three different quartile computation algorithms.
 
 ```python
 import plotly.express as px
@@ -103,7 +104,7 @@ df = pd.DataFrame(dict(
     linear=data,
     inclusive=data,
     exclusive=data
-)).melt(var_name="quartilemethod") 
+)).melt(var_name="quartilemethod")
 
 
 fig = px.box(df, y="value", facet_col="quartilemethod", color="quartilemethod",
@@ -204,7 +205,7 @@ fig.show()
 
 You can specify precomputed quartile attributes rather than using a built-in quartile computation algorithm.
 
-This could be useful if you have already pre-computed those values or if you need to use a different algorithm than the ones provided. 
+This could be useful if you have already pre-computed those values or if you need to use a different algorithm than the ones provided.
 
 ```python
 import plotly.graph_objects as go
@@ -217,9 +218,9 @@ fig.add_trace(go.Box(y=[
         [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]
       ], name="Precompiled Quartiles"))
 
-fig.update_traces(q1=[ 1, 2, 3 ], median=[ 4, 5, 6 ], 
-                  q3=[ 7, 8, 9 ], lowerfence=[-1, 0, 1], 
-                  upperfence=[5, 6, 7], mean=[ 2.2, 2.8, 3.2 ], 
+fig.update_traces(q1=[ 1, 2, 3 ], median=[ 4, 5, 6 ],
+                  q3=[ 7, 8, 9 ], lowerfence=[-1, 0, 1],
+                  upperfence=[5, 6, 7], mean=[ 2.2, 2.8, 3.2 ],
                   sd=[ 0.2, 0.4, 0.6 ], notchspan=[ 0.2, 0.4, 0.6 ] )
 
 fig.show()

doc/python/bubble-charts.md

@@ -38,7 +38,7 @@ jupyter:
 
 A [bubble chart](https://en.wikipedia.org/wiki/Bubble_chart) is a scatter plot in which a third dimension of the data is shown through the size of markers. For other types of scatter plot, see the [line and scatter page](https://plot.ly/python/line-and-scatter/).
 
-We first show a bubble chart example using Plotly Express. [Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/). The size of markers is set from the dataframe column given as the `size` parameter.
+We first show a bubble chart example using Plotly Express. [Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/) and produces [easy-to-style figures](/python/styling-plotly-express/). The size of markers is set from the dataframe column given as the `size` parameter.
 
 ```python
 import plotly.express as px

doc/python/bubble-maps.md

@@ -39,7 +39,7 @@ Plotly figures made with `px.scatter_geo`, `px.line_geo` or `px.choropleth` func
 
 ### Bubble map with Plotly Express
 
-[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/). With `px.scatter_geo`, each line of the dataframe is represented as a marker point. The column set as the `size` argument gives the size of markers.
+[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/) and produces [easy-to-style figures](/python/styling-plotly-express/). With `px.scatter_geo`, each line of the dataframe is represented as a marker point. The column set as the `size` argument gives the size of markers.
 
 ```python
 import plotly.express as px

doc/python/choropleth-maps.md

@@ -56,7 +56,7 @@ The GeoJSON data is passed to the `geojson` argument, and the data is passed int
 
 ### Choropleth Map with plotly.express
 
-[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/).
+[Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/) and produces [easy-to-style figures](/python/styling-plotly-express/).
 
 #### GeoJSON with `feature.id`
 

doc/python/datashader.md

@@ -0,0 +1,129 @@
+---
+jupyter:
+  jupytext:
+    notebook_metadata_filter: all
+    text_representation:
+      extension: .md
+      format_name: markdown
+      format_version: "1.2"
+      jupytext_version: 1.3.1
+  kernelspec:
+    display_name: Python 3
+    language: python
+    name: python3
+  language_info:
+    codemirror_mode:
+      name: ipython
+      version: 3
+    file_extension: .py
+    mimetype: text/x-python
+    name: python
+    nbconvert_exporter: python
+    pygments_lexer: ipython3
+    version: 3.6.8
+  plotly:
+    description:
+      How to use datashader to rasterize large datasets, and visualize
+      the generated raster data with plotly.
+    display_as: scientific
+    language: python
+    layout: base
+    name: Plotly and Datashader
+    order: 21
+    page_type: u-guide
+    permalink: python/datashader/
+    thumbnail: thumbnail/datashader.jpg
+---
+
+[datashader](https://datashader.org/) creates rasterized representations of large datasets for easier visualization, with a pipeline approach consisting of several steps: projecting the data on a regular grid, creating a color representation of the grid, etc.
+
+### Passing datashader rasters as a mabox image layer
+
+We visualize here the spatial distribution of taxi rides in New York City. A higher density
+is observed on major avenues. For more details about mapbox charts, see [the mapbox layers tutorial](/python/mapbox-layers). No mapbox token is needed here.
+
+```python
+import pandas as pd
+df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/uber-rides-data1.csv')
+dff = df.query('Lat < 40.82').query('Lat > 40.70').query('Lon > -74.02').query('Lon < -73.91')
+
+import datashader as ds
+cvs = ds.Canvas(plot_width=1000, plot_height=1000)
+agg = cvs.points(dff, x='Lon', y='Lat')
+# agg is an xarray object, see http://xarray.pydata.org/en/stable/ for more details
+coords_lat, coords_lon = agg.coords['Lat'].values, agg.coords['Lon'].values
+# Corners of the image, which need to be passed to mapbox
+coordinates = [[coords_lon[0], coords_lat[0]],
+               [coords_lon[-1], coords_lat[0]],
+               [coords_lon[-1], coords_lat[-1]],
+               [coords_lon[0], coords_lat[-1]]]
+
+from colorcet import fire
+import datashader.transfer_functions as tf
+img = tf.shade(agg, cmap=fire)[::-1].to_pil()
+
+import plotly.express as px
+# Trick to create rapidly a figure with mapbox axes
+fig = px.scatter_mapbox(dff[:1], lat='Lat', lon='Lon', zoom=12)
+# Add the datashader image as a mapbox layer image
+fig.update_layout(mapbox_style="carto-darkmatter",
+                 mapbox_layers = [
+                {
+                    "sourcetype": "image",
+                    "source": img,
+                    "coordinates": coordinates
+                }]
+)
+fig.show()
+```
+
+### Exploring correlations of a large dataset
+
+Here we explore the flight delay dataset from https://www.kaggle.com/usdot/flight-delays. In order to get a visual impression of the correlation between features, we generate a datashader rasterized array which we plot using a `Heatmap` trace. It creates a much clearer visualization than a scatter plot of (even a fraction of) the data points, as shown below.
+
+Note that instead of datashader it would theoretically be possible to create a [2d histogram](/python/2d-histogram-contour/) with plotly but this is not recommended here because you would need to load the whole dataset (5M rows !) in the browser for plotly.js to compute the heatmap, which is practically not tractable. Datashader offers the possibility to reduce the size of the dataset before passing it to the browser.
+
+```python
+import plotly.graph_objects as go
+import pandas as pd
+import numpy as np
+import datashader as ds
+df = pd.read_parquet('https://raw.githubusercontent.com/plotly/datasets/master/2015_flights.parquet')
+fig = go.Figure(go.Scattergl(x=df['SCHEDULED_DEPARTURE'][::200],
+                             y=df['DEPARTURE_DELAY'][::200],
+                             mode='markers')
+)
+fig.update_layout(title_text='A busy plot')
+fig.show()
+```
+
+```python
+import plotly.graph_objects as go
+import pandas as pd
+import numpy as np
+import datashader as ds
+df = pd.read_parquet('https://raw.githubusercontent.com/plotly/datasets/master/2015_flights.parquet')
+
+cvs = ds.Canvas(plot_width=100, plot_height=100)
+agg = cvs.points(df, 'SCHEDULED_DEPARTURE', 'DEPARTURE_DELAY')
+x = np.array(agg.coords['SCHEDULED_DEPARTURE'])
+y = np.array(agg.coords['DEPARTURE_DELAY'])
+
+# Assign nan to zero values so that the corresponding pixels are transparent
+agg = np.array(agg.values, dtype=np.float)
+agg[agg<1] = np.nan
+
+fig = go.Figure(go.Heatmap(
+    z=np.log10(agg), x=x, y=y,
+    hoverongaps=False,
+    hovertemplate='Scheduled departure: %{x:.1f}h <br>Depature delay: %{y} <br>Log10(Count): %{z}',
+    colorbar=dict(title='Count (Log)', tickprefix='1.e')))
+fig.update_xaxes(title_text='Scheduled departure')
+fig.update_yaxes(title_text='Departure delay')
+fig.show()
+
+```
+
+```python
+
+```

doc/python/distplot.md

@@ -37,7 +37,7 @@ jupyter:
 
 Several representations of statistical distributions are available in plotly, such as [histograms](https://plot.ly/python/histograms/), [violin plots](https://plot.ly/python/violin/), [box plots](https://plot.ly/python/box-plots/) (see [the complete list here](https://plot.ly/python/statistical-charts/)). It is also possible to combine several representations in the same plot.
 
-For example, the `plotly.express` function `px.histogram` can add a subplot with a different statistical representation than the histogram, given by the parameter `marginal`. [Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/).
+For example, the `plotly.express` function `px.histogram` can add a subplot with a different statistical representation than the histogram, given by the parameter `marginal`. [Plotly Express](/python/plotly-express/) is the easy-to-use, high-level interface to Plotly, which [operates on "tidy" data](/python/px-arguments/) and produces [easy-to-style figures](/python/styling-plotly-express/).
 
 ```python
 import plotly.express as px