bitmap output

[oroszl]

Is there a way for producing a bitmap image of the rendnering ? Looking at some threejs examples I found that from threejs it is in principle possible to produce a png output, but if I am not mistaken pythreejs is not yet setup to do so.

[jasongrout]

Yep, there are ways to do it, but we are not set up to do it yet. We'll keep this issue open as a todo item.

[oroszl]

Thanks..

[oroszl]

from this post
http://stackoverflow.com/questions/15558418/how-do-you-save-an-image-from-a-three-js-canvas
it seems that if one uses the canvas renderer threejs would be able to generate a dataurl.
is it hard to bubble it up to pythreejs ?

[jasongrout]

no, not hard at all. Or we can do it with the webgl renderer, as in the first answer.

[jasongrout]

This also looks interesting: http://stackoverflow.com/a/22579139

[oroszl]

I was also wondering if it is possible to use ipython it self maybe trough some trickery with display() ?

[arizhakov]

Has there been any updates on this issue (e.g., concrete pythreejs implementation of three.js capability to produce image data output), or are the above links the only leads to play with at the moment?

[MannyKayy]

Bumping this issue

[maartenbreddels]

In ipyvolume I do screenshots by indeed saving the canvas to a png, and sending that over the wire. I think a good way so solve this issue (and #167) is to make the renderer an (ipywebrtc) MediaStream, and put this feature in ipywebrtc. The video part is already working, I think it's not so difficult to add what is done on the backend and frontend to ipywebrtc (frontend and backend)

Pythreejs's renderer needs to inherit from from MediaStream (backend) and deliver a mediastream (for the first view) on the frontend.
If someone wants to take a shot, I'm happy to help.

[elPistolero]

With help from the code (backend and frontend) I implemented a first working version.

I haven't figured out how to directly return the image data from the RenderableWidget screenshot method yet.

[maartenbreddels]

There is a gory way of doing that, if you start diving into this part which ends up doing a busy loop
If you find a better way, let me know.
In any case, I think this should all move into ipywebrtc/jupyter-webrtc

[elPistolero]

I saw the busy loop, but I was wondering whether it would be possible for the screenshot function itself to internally wait until the data is ready and then return the image.
So something like:

def screenshot(self):
        self.screenshot_ready = False
        content = {
            "type": "screenshot"
        }
        self.send(content)
        busy_loop()
        return self.screenshot_data

However, when I tried this the returned data seems to be gobbled up.
Does this have something to do with the fact that RenderableWidget is a DOMWidget?

Is moving this into ipywebrtc/jupyter-webrtc and introducing a new dependency for pythreejs something the maintainers of this project want?

[maartenbreddels]

I haven't seen gobbled up data, but I have seen the output coming after that being all over the place. That's why I use with output: a lot.
@vidartf you are ok with having a ipywebrtc dependency right? +making the Renderer inherit from ipywebrtc.MediaStream ?

[vidartf]

I'm a lot more concerned about the PIL dependency than about the ipywebrtc one, but hopefully using ipywebrtc can solve that.

I'm all for having a way to render to a ipywebrtc.MediaStream. Whether that should be the default for all Renderer instances is another matter (I would maybe make a separate class StreamRenderer to manage this, but we should go over the different use cases in detail).

I'm also concerned about the do_one_iteration() code. I've worked with that before and seem to remember that there are some scenarios that will cause that pattern to hang the kernel. I will have to refresh my memory of this though, so feel free to disregard this for now.

[jsb]

As a workaround, I copied ipyvolume's JavaScript code for saving screenshots in the browser and added a Python method to trigger this (based on @elPistolero's code). The modified code is here.

I use the code like this:

for i in range(k):
    # prepare and display scene i
    time.sleep(0.5)
    renderer.download_screenshot('screenshot-{}.png'.format(i))
    clear_output()

Results are super flimsy: It seems to only work with the delay between display and the call to download_screenshot. If I set the delay too low it will skip some images. Of course increasing the delay slows down the overall batch process. Since the images are downloaded through the browser, I need to set it up to download images without confirmation unless I want to click the confirmation dialog several times. It's frustrating to work with but currently the only way that lets me batch produce screenshots from pythreejs.

[dimatura]

In the "(really) hacky workarounds" category, a method I use that doesn't require modification of pythreejs is to grab the canvas from the jupyter notebook DOM with javascript injection, and pass the output to python with Jupyter.notebook.kernel.execute. Like the above methods it does require a delay or busy waiting, and in my experience it's kind of flaky.

That is, assuming you only have one pythreejs canvas, use IPython.display.Javascript (or the %%javascript magic) to run something like

var canvas = $('canvas')[0];
var data = canvas.toDataURL();
Jupyter.notebook.kernel.execute("canvas_data = '" + data + "'");

Then after a few milliseconds, canvas_data will have the base64-encoded PNG, which can be directly saved to a file (after base64-decoding) or turned into a PIL image:

png_data = base64.decodestring(canvas_data[21:])
pil_img = Image.open(StringIO(png_data))

This is slightly more convenient, I think, than the above method, but still a big hack, of course.

[vidartf]

The current long term plan here is to try to integrate ipywebrtc into pythreejs. I think I will try to bug @maartenbreddels a bit to help me get started though ;)

[maartenbreddels]

https://ipywebrtc.readthedocs.io/en/latest/WidgetStream.html + https://ipywebrtc.readthedocs.io/en/latest/ImageRecorder.html should allow you to capture images, the first demos capturing a movie, but if you stick the WidgetStream into the ImageRecorder, you should be set.

[vidartf]

Sure, I was just thinking there would be some possible optimizations by integrating it fully instead of relying on WidgetStream (?)

[TakodaS]

ThreeJS has the method SVGRenderer that does exactly this. Pythreejs should have a wrapper for this.