Failed to start Jupyter service with configurable-http-proxy 4.4.0

[MvdB202]

Bug description

With configurable-http-proxy version 4.4.0, JupyterHub failed to start proxy. Output:

/usr/lib/node_modules/configurable-http-proxy/node_modules/prom-client/lib/metrics/osMemoryHeapLinux.js:57
			} catch {
			        ^

SyntaxError: Unexpected token {
    at createScript (vm.js:80:10)
    at Object.runInThisContext (vm.js:139:10)
    at Module._compile (module.js:617:28)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/usr/lib/node_modules/configurable-http-proxy/node_modules/prom-client/lib/metrics/osMemoryHeap.js:4:22) 

Expected behaviour

I expected JupyterHub service to start as usual with the config proxy adding default route for Hub > http://127.0.0.1:8081

Actual behaviour

May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.563 JupyterHub proxy:666] Starting proxy @ http://:8000

How to reproduce

My instance is created with an ansible playbook where I had the npm module install configurable-http-proxy globally. Which now uses version 4.4.0 by default, which triggered this bug.
Currently I have version: '4.3.2' added to my playbook and everything is back up and running.

Your personal set up

jupyterhub version 1.3.0
Python 3.6.9
Ubuntu 18.04
nginx 1.14.0

• Configuration

# jupyterhub_config.py
c.Spawner.default_url = '/lab?reset'
c.Spawner.notebook_dir = '~'
c.Authenticator.admin_users = {'ubuntu'}

• Logs

# paste relevant logs here, if any
May 31 10:37:46 my-instance systemd[1]: Started Jypyter Hub daemon.
May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.507 JupyterHub app:2349] Running JupyterHub version 1.3.0
May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.508 JupyterHub app:2380] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.3.0
May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.508 JupyterHub app:2380] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-1.3.0
May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.508 JupyterHub app:2380] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0
May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.510 JupyterHub app:1420] Loading cookie_secret from /jupyterhub_cookie_secret
May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.524 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.546 JupyterHub app:1726] Not using allowed_users. Any authenticated user will be allowed.
May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.559 JupyterHub app:2419] Initialized 0 spawners in 0.001 seconds
May 31 10:37:46 my-instance jupyterhub[14230]: [W 2021-05-31 10:37:46.561 JupyterHub proxy:529] Found proxy pid file: /jupyterhub-proxy.pid
May 31 10:37:46 my-instance jupyterhub[14230]: [W 2021-05-31 10:37:46.562 JupyterHub proxy:541] Proxy no longer running at pid=14214
May 31 10:37:46 my-instance jupyterhub[14230]: [W 2021-05-31 10:37:46.562 JupyterHub proxy:663] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
May 31 10:37:46 my-instance jupyterhub[14230]: [I 2021-05-31 10:37:46.563 JupyterHub proxy:666] Starting proxy @ http://:8000
May 31 10:37:46 my-instance jupyterhub[14230]: [E 2021-05-31 10:37:46.567 JupyterHub utils:80] Unexpected error connecting to my-instance:8000 [Errno 22] Invalid argument
May 31 10:37:46 my-instance jupyterhub[14230]: [E 2021-05-31 10:37:46.622 JupyterHub utils:80] Unexpected error connecting to my-instance:8000 [Errno 22] Invalid argument
May 31 10:37:46 my-instance jupyterhub[14230]: /usr/lib/node_modules/configurable-http-proxy/node_modules/prom-client/lib/metrics/osMemoryHeapLinux.js:57
May 31 10:37:46 my-instance jupyterhub[14230]:                         } catch {
May 31 10:37:46 my-instance jupyterhub[14230]:                                 ^
May 31 10:37:46 my-instance jupyterhub[14230]: SyntaxError: Unexpected token {
May 31 10:37:46 my-instance jupyterhub[14230]:     at createScript (vm.js:80:10)
May 31 10:37:46 my-instance jupyterhub[14230]:     at Object.runInThisContext (vm.js:139:10)
May 31 10:37:46 my-instance jupyterhub[14230]:     at Module._compile (module.js:617:28)
May 31 10:37:46 my-instance jupyterhub[14230]:     at Object.Module._extensions..js (module.js:664:10)
May 31 10:37:46 my-instance jupyterhub[14230]:     at Module.load (module.js:566:32)
May 31 10:37:46 my-instance jupyterhub[14230]:     at tryModuleLoad (module.js:506:12)
May 31 10:37:46 my-instance jupyterhub[14230]:     at Function.Module._load (module.js:498:3)
May 31 10:37:46 my-instance jupyterhub[14230]:     at Module.require (module.js:597:17)
May 31 10:37:46 my-instance jupyterhub[14230]:     at require (internal/module.js:11:18)
May 31 10:37:46 my-instance jupyterhub[14230]:     at Object.<anonymous> (/usr/lib/node_modules/configurable-http-proxy/node_modules/prom-client/lib/metrics/osMemoryHeap.js:4:22)
May 31 10:37:46 my-instance jupyterhub[14230]: [E 2021-05-31 10:37:46.758 JupyterHub utils:80] Unexpected error connecting to my-instance:8000 [Errno 22] Invalid argument
May 31 10:37:47 my-instance jupyterhub[14230]: [E 2021-05-31 10:37:47.406 JupyterHub utils:80] Unexpected error connecting to my-instance:8000 [Errno 22] Invalid argument
May 31 10:37:47 my-instance jupyterhub[14230]: [E 2021-05-31 10:37:47.662 JupyterHub utils:80] Unexpected error connecting to my-instance:8000 [Errno 22] Invalid argument
May 31 10:37:47 my-instance jupyterhub[14230]: [C 2021-05-31 10:37:47.662 JupyterHub app:2625] Failed to start proxy
May 31 10:37:47 my-instance jupyterhub[14230]:     Traceback (most recent call last):
May 31 10:37:47 my-instance jupyterhub[14230]:       File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 2623, in start
May 31 10:37:47 my-instance jupyterhub[14230]:         await self.proxy.start()
May 31 10:37:47 my-instance jupyterhub[14230]:       File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 693, in start
May 31 10:37:47 my-instance jupyterhub[14230]:         _check_process()
May 31 10:37:47 my-instance jupyterhub[14230]:       File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 689, in _check_process
May 31 10:37:47 my-instance jupyterhub[14230]:         raise e from None
May 31 10:37:47 my-instance jupyterhub[14230]:     RuntimeError: Proxy failed to start with exit code 1
May 31 10:37:47 my-instance jupyterhub[14230]:     
May 31 10:37:47 my-instance systemd[1]: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE
May 31 10:37:47 my-instance systemd[1]: jupyterhub.service: Failed with result 'exit-code'.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[consideRatio]

Thanks for reporting this bug, syntax error it sais - i must say i love syntax error bugs compared to all other bugs!!! ❤️

[Icare2000]

I have the same issue!

[ErickSystem]

@Icare2000 and @MvdB202, I came back from version latest (published 7 days ago: 4.4.0) to npm install -g [email protected], my issue was solved.

[manics]

Hi! If you run configurable-http-proxy directly do you still see this error? If so could you give us a full reproducible workflow that we could run in e.g. a fresh virtual machine from scratch?

[MvdB202]

Hi! If you run configurable-http-proxy directly do you still see this error? If so could you give us a full reproducible workflow that we could run in e.g. a fresh virtual machine from scratch?

If I run it directly I see the output I put in the top of my issue under bug description, I just started a new VM, manually updated 4.3.2 to 4.4.0, restarted my jupyterhub.service and get the same issue. My Jupyter webconsole turns from being accessible to a 502 Bad Gateway message.

My workflow: with ansible I spin up a VM with Ubuntu 18.04, nginx 1.14.0, python 3.6.9, then I install nodejs, jupyterlab 3.0.11, jupyterhub 1.3.0, virtualenv 20.4.3, jupyterlab-git 0.30.0b3 and configurable-http-proxy (used to not specify version, but playbook now specifies 4.3.2). My nginx config for jupyterhub ( /etc/nginx/app-location-conf.d/jupyterhub.conf):

location / {
    proxy_pass http://127.0.0.1:8000;
    proxy_redirect http://localhost:8000/ $scheme://$host/;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    client_max_body_size 10G;
}

My /etc/jupyterhub/jupyterhub_config.py :

c.Spawner.default_url = '/lab?reset'
c.Spawner.notebook_dir = '~'
c.Authenticator.admin_users = {'ubuntu'}

My /usr/lib/systemd/system/jupyterhub.service :

[Unit]
Description=Jypyter Hub daemon
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/jupyterhub --config=/etc/jupyterhub/jupyterhub_config.py
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

Then I upgrade nbdime and start/enable jupyterhub.service.

[consideRatio]

@MvdB202 what version of node is installed (node --version)? configurable-http-proxy is a node application and that have been relevant in the past.

It could be relevant to know the version of the prom-client library used as well, but I don't know how to get that information.

@dtaniwaki do you have a guess on how to resolve this?

[MvdB202]

@MvdB202 what version of node is installed (node --version)? configurable-http-proxy is a node application and that have been relevant in the past.

It could be relevant to know the version of the prom-client library used as well, but I don't know how to get that information.

Oh yes sorry, node version v8.17.0. I did an npm -g list with a grep on prom-client but no result.

[consideRatio]

@MvdB202 ah! That is likely an issue! Can you see if upgrading to node 12 or later resolves the issue?

configurable-http-proxy seem to be failing to require a certain node version, we only test against 10+ and as node 10 is no longer maintained and I think we should enforce node 12 and higher at this point in time. I've seen node 10 tests fail intermittently in our tests as well, and I know old versions of node has caused trouble in the past for us.

configurable-http-proxy/.github/workflows/test.yml

Lines 65 to 69 in 790a8c7

	node_version:
	- "10"
	- "12"
	- "14"
	- "15"

[Icare2000]

I also tested on my system. Node was outdated. Now it's working with 14!

[dtaniwaki]

Sorry for the late response. I agree that we should follow the maintained node versions and test against them in the CI!

[MvdB202]

@MvdB202 ah! That is likely an issue! Can you see if upgrading to node 12 or later resolves the issue?

configurable-http-proxy seem to be failing to require a certain node version, we only test against 10+ and as node 10 is no longer maintained and I think we should enforce node 12 and higher at this point in time. I've seen node 10 tests fail intermittently in our tests as well, and I know old versions of node has caused trouble in the past for us.

configurable-http-proxy/.github/workflows/test.yml

Lines 65 to 69 in 790a8c7

	node_version:
	- "10"
	- "12"
	- "14"
	- "15"

@consideRatio You were right! I did not think to check the node version before, or thought that was the issue, but I just tested this and it works! Thanks for your help, I will close the topic.