Cannot proxy websocket from Gtk Broadway server

[paulmthompson]

Bug description

I am using Jupyterhub to spawn arbitrary images, and one uses the program Broadway to display Gtk GUIs in browsers. For some reason, CHP is not able to proxy the websocket communication from this program.
https://developer.gnome.org/gtk3/stable/gtk-broadway.html
For all of my testing below, I'll just use CHP directly without JH for simplicity.

Expected behaviour

I can open a broadway server on port 8081 and draw a GUI there. I can then set up a proxy with CHP on port 8600 and view the GUI on that port.

Actual behaviour

I can view the GUI on 8081, but not on 8600. The first part of the http communication seems to work correctly because the firefox tab for 8600 says "broadway 2.0", but no GUI is ever rendered. If I go to the web dev console in firefox it shows a 400 websocket error.

I don't think that this is a broadway problem because I can set up a reverse proxy without problems using NGINX with a websocket configuration.

How to reproduce

Download Gtk. Open a terminal, and start broadway on port 8081 with

broadwayd :1

In a different terminal, create a proxy with CHP on port 8600:

configurable-http-proxy --default-target=http://127.0.0.1:8081 --ip localhost --port 8600

Your personal set up

• OS:
  Ubuntu 18.04
• Version:
  CHP version 4.2.1.
  Node version 14.4.0

It is possible this is some broadway quirk, but I'm not sure why it doesn't happen with NGINX. If I look at the header in the firefox web console, the requests appear to be identical to both the NGINX and CHP proxies, however the (working) NGINX one shows a status code of "101 switching protocols" instead of "400 bad websocket request". NGINX also shows Version HTTP/1.1, but CHP shows Version HTTP/1.0 (this seems bad?)

Any ideas on next steps would be great appreciated. Let me know if you need any additional information or testing.

[consideRatio]

@paulmthompson I had no idea what broadway was but apparently it was a CLI broadway already available to me on Ubuntu 20.04 so I tried to reproduce what you described - success, issue confirmed.

I added --log-level=debug to the startup of CHP, and could see the following after makinga request to localhost:8600 using my browser.

(base) erik@sos:~/dev/jupyterhub/configurable-http-proxy/bin$ ./configurable-http-proxy --default-target=http://127.0.0.1:8081 --ip localhost --port 8600 --log-level=debug
21:16:19.004 [ConfigProxy] warn: REST API is not authenticated.
21:16:19.010 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
21:16:19.015 [ConfigProxy] info: Proxying http://localhost:8600 to http://127.0.0.1:8081
21:16:19.015 [ConfigProxy] info: Proxy API at http://localhost:8601/api/routes
21:16:19.016 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
21:16:34.090 [ConfigProxy] debug: PROXY WEB / to http://127.0.0.1:8081
21:16:34.199 [ConfigProxy] debug: PROXY WEB /broadway.js to http://127.0.0.1:8081
21:16:34.224 [ConfigProxy] debug: PROXY WS /socket to http://127.0.0.1:8081
21:16:34.274 [ConfigProxy] debug: PROXY WEB /rawinflate.min.js.map to http://127.0.0.1:8081

Inspecting the request I made using chrome inspector tools, I saw that I made a http request that succeeded to localhost, and got a almost blank HTML response that refererred to broadway.js function called connect. This in turn tried to open a connection.

A guess that was wrong

I think the crux may be this line in the logs, a usage of http:// when arriving as ws://?

21:16:34.224 [ConfigProxy] debug: PROXY WS /socket to http://127.0.0.1:8081

UPDATE: No this was not the case, this was just the log message and by modifying things it seems like it didn't matter.

To debug further

CHP depends on http-party/node-http-proxy to do its work, so in the end its how we make use of it that can be the issue, or the issue could be part of node-http-proxy itself. I'm not sure.

Here is some related documentation in node-http-proxy. I have confirmed that CHP configures the option called ws to be true.

I struggled for some hours to resolve this but failed =/ I created a PR (#248) with some inline doc and a minor unrelated fix to something nobody probably cares about.

[paulmthompson]

Thanks for your reply!

Other failed attempted solutions

I have also tried for some time without success. I experimented directly with node-http-proxy as well, and I can confirm that node-http-proxy seems unable to create a barebones websocket proxy for broadway. Let me document a few things I tried:

I think sometimes a proxy can modify and mess up headers for the request, so using SSL can let them pass through unchanged since they can't be read and modified. I tried using SSL and still got the same error.
https://stackoverflow.com/questions/38974731/websockets-not-connected-behind-proxy

I included code in the simple node proxy server to make sure that upgrade requests are handled as websockets. I did not see any change.
https://stackoverflow.com/questions/21629752/using-node-http-proxy-to-proxy-websocket-connections

Ideas for future debugging

The "proxy_pass" directive in NGINX seems to work fine with broadway. Here is a pretty decent explanation of what it does to headers when it gets a request. Maybe there is something that node is not doing?
https://www.digitalocean.com/community/tutorials/understanding-nginx-http-proxying-load-balancing-buffering-and-caching

Workaround

I can spawn GTK GUIs (or any X GUI) with jupyterhub by drawing into an xpra x11 bridge. No broadway required.
https://github.com/JAremko/docker-x11-bridge

I then use one additional container with a nginx reverse proxy so that xxx.xxx.xxx/user/paul proxies to the xpra server.

Other thoughts

I have an orphan SO question about this here :(
https://stackoverflow.com/questions/62327427/can-reverse-proxy-a-gtk-broadway-server-with-nginx-but-not-with-node-http-proxy

This issue is probably more appropriate in http-party/node-http-proxy since I can reproduce the bug there. The workaround above also works well enough for my purposes. I'm fine with closing the issue here if we are out of ideas for now.

[consideRatio]

This issue is probably more appropriate in http-party/node-http-proxy since I can reproduce the bug there. The workaround above also works well enough for my purposes. I'm fine with closing the issue here if we are out of ideas for now.

Ah nice that you can reproduce it with node-http-proxy in isolation! Please link back to this issue if you open an issue there.

Good luck Paul!

[consideRatio]

I'll go ahead and close this issue as out of scope for CHP to address.