Add ipywebrtc captureStream interface

examples/Capture.ipynb

@@ -0,0 +1,289 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Capture outputs\n",
+    "\n",
+    "This notebook will demonstrate how to capture still frames or videos from pythreejs using [ipywebrtc](https://ipywebrtc.readthedocs.io/en/latest/)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup an example renderer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pythreejs import *\n",
+    "import ipywebrtc\n",
+    "from ipywidgets import Output, VBox"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "view_width = 600\n",
+    "view_height = 400\n",
+    "\n",
+    "sphere = Mesh(\n",
+    "    SphereBufferGeometry(1, 32, 16),\n",
+    "    MeshStandardMaterial(color='red')\n",
+    ")\n",
+    "\n",
+    "cube = Mesh(\n",
+    "    BoxBufferGeometry(1, 1, 1),\n",
+    "    MeshPhysicalMaterial(color='green'),\n",
+    "    position=[2, 0, 4]\n",
+    ")\n",
+    "\n",
+    "camera = PerspectiveCamera( position=[10, 6, 10], aspect=view_width/view_height)\n",
+    "key_light = DirectionalLight(position=[0, 10, 10])\n",
+    "ambient_light = AmbientLight()\n",
+    "\n",
+    "scene = Scene(children=[sphere, cube, camera, key_light, ambient_light])\n",
+    "controller = OrbitControls(controlling=camera)\n",
+    "renderer = Renderer(camera=camera, scene=scene, controls=[controller],\n",
+    "                    width=view_width, height=view_height)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "renderer"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Capture renderer output to stream"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "stream = ipywebrtc.WidgetStream(widget=renderer, max_fps=30)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If you want, you can preview the content of the stream with a video-viewer. This should simply mirror what you see in the renderer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": false
+   },
+   "outputs": [],
+   "source": [
+    "stream"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Capturing images\n",
+    "\n",
+    "To capture images from the stream, use the `ImageRecorder` widget from `ipywebrtc`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "recorder = ipywebrtc.ImageRecorder(filename='snapshot', format='png', stream=stream)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "There are two ways to capture images from the stream:\n",
+    "1. Manually from the browser by using the widget view of the recorder.\n",
+    "2. Programmatically using the .save()/download() method on the recorder."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Using the view"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "recorder"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here,clicking the \"Snapshot\" button will capture a new frame and sync it back to the kernel side. Clicking \"Download\" will download the current snapshot on the *client side*. When taking a snapshot, the image will also be synced to the *kernel side*. If the image has changed, any observers of the value trait of the image will trigger (i.e. `recorder.image.observe(callback, 'value')`):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "out = Output()  # To capture print output\n",
+    "\n",
+    "@out.capture()\n",
+    "def on_capture(change):\n",
+    "    print('Captured image changed!')\n",
+    "recorder.image.observe(on_capture, 'value')\n",
+    "out"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Using kernel API:\n",
+    "\n",
+    "To request a snapshot from the kernel, set the `recording` attribute of the recorder to `True`. This will update the `image` attribute asynchronously. The easiest way to save this to the kernel side is to also set the `filename` attribute, and set `autosave` to `True`. This will cause the image to be saved as soon as it is available. This is equivalend to observing the image widget's `value` trait, and calling the `save()` method when the image changes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "recorder.autosave = True\n",
+    "recorder.recording = True"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also trigger a client-side download from the kernel by calling the `download()` method on the recorder:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "recorder.download()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Capturing video\n",
+    "\n",
+    "To capture a video from the stream, use the `VideoRecorder` from `ipywebrtc`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "video_recorder = ipywebrtc.VideoRecorder(stream=stream, filename='video', codecs='vp8')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "video_recorder"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here, clicking the \"Record\" button will start capturing the video. Once you click the \"Stop\" button (appears after clicking \"Record\"), the video will be displayed in the view, and it will be synced to the kernel. If the video has changed, any observers of the value trait of the video will trigger, similarly to that of the `ImageRecorder`. Clicking \"Download\" will download the current video on the client side.\n",
+    "\n",
+    "The kernel side API for the `VideoRecorder` is similar to that of the `ImageRecorder`, but you will also have to tell it when to stop:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "video_recorder.autosave = True\n",
+    "video_recorder.recording = True\n",
+    "# After executing this, try to interact with the renderer above before executing the next cell"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "video_recorder.recording = False\n",
+    "video_recorder.download()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

js/package.json

@@ -20,7 +20,8 @@
     "build:labextension": "rimraf lab-dist && mkdirp lab-dist && cd lab-dist && npm pack ..",
     "build:all": "npm run build:labextension",
     "prepare": "npm run autogen",
-    "prepack": "npm run build:bundles-prod"
+    "prepack": "npm run build:bundles-prod",
+    "watch": "webpack -d -w"
   },
   "devDependencies": {
     "eslint": "^5.6.0",

js/src/_base/Renderable.js

@@ -1,6 +1,7 @@
 var _ = require('underscore');
 var widgets = require('@jupyter-widgets/base');
 var $ = require('jquery');
+var Promise = require('bluebird');
 
 var pkgName = require('../../package.json').name;
 var EXTENSION_SPEC_VERSION = require('../version').EXTENSION_SPEC_VERSION;
@@ -90,6 +91,83 @@ var RenderableModel = widgets.DOMWidgetModel.extend({
         this.trigger('childchange', this);
     },
 
+    /**
+     * Find a view, preferrably a live one
+     */
+    _findView: function() {
+        var viewPromises = Object.keys(this.views).map(function(key) {
+            return this.views[key];
+        }, this);
+        return Promise.all(viewPromises).then(function(views) {
+            for (var i=0; i<views.length; ++i) {
+                var view = views[i];
+                if (!view.isFrozen) {
+                    return view;
+                }
+            }
+            return views[0];
+        });
+    },
+
+    /**
+     * Interface for jupyter-webrtc.
+     */
+    captureStream: function(fps) {
+        var stream = new MediaStream();
+
+        var that = this;
+        var canvasStream = null;
+
+        function updateStream() {
+            return that._findView().then(function(view) {
+                if (canvasStream !== null) {
+                    // Stop and remove tracks from previous canvas
+                    stream.getTracks().forEach(function(track) {
+                        track.stop();
+                        stream.removeTrack(track);
+                        canvasStream.removeTrack(track);
+                    });
+                    canvasStream = null;
+                }
+                var canvas;
+                if (view.isFrozen) {
+                    canvas = document.createElement('canvas');
+                    canvas.width = view.$frozenRenderer.width();
+                    canvas.height = view.$frozenRenderer.height();
+                    var ctx = canvas.getContext('2d');
+                    ctx.drawImage(view.$frozenRenderer[0], 0, 0);
+                } else {
+                    canvas = view.renderer.domElement;
+                }
+                // Add tracks from canvas to stream
+                canvasStream = canvas.captureStream(fps);
+                canvasStream.getTracks().forEach(function(track) {
+                    stream.addTrack(track);
+                    if (track.requestFrame) {
+                        (function() {
+                            var orig = track.requestFrame.bind(track);
+                            track.requestFrame = function() {
+                                orig();
+                                // Ensure we redraw to make stream pickup first frame on Chrome
+                                // https://bugs.chromium.org/p/chromium/issues/detail?id=903832
+                                view.tick();
+                            };
+                            track.requestFrame();
+
+                        }());
+                    }
+                });
+
+                // If renderer status changes, update stream
+                that.listenToOnce(view, 'updatestream', updateStream);
+            });
+        }
+
+        return updateStream().then(function() {
+            return stream;
+        });
+    },
+
 }, {
     serializers: _.extend({
         clippingPlanes: { deserialize: unpackThreeModel },
@@ -199,6 +277,7 @@ var RenderableView = widgets.DOMWidgetView.extend({
         } else {
             this.renderer.setSize(width, height);
         }
+        this.trigger('updatestream');
     },
 
     updateProperties: function(force) {