[Bug]: code-server hang

[ericzhucode]

Is there an existing issue for this?

• I have searched the existing issues

I know this issue may not be caused by code-server but nodejs since when process hangs, it seems no any js code is running and all nodejs environment stucks. Just post this here to see if it's possible to get professional suggestions from your team.
I will also post this issue at nodejs community
Thank you for your team's effort!

OS/Web Information

• Web Browser: Chrome/Edge(This issue is only related to backend process on Linux system)
• Local OS: Win10/Win11
• Remote OS: Linux CentOS
• Remote Architecture:
• code-server --version: 4.1.0 / 4.2.0 / 4.4.0 (Some customers are upgrading to 4.5.3 now and we will monitor if this version also has the issue)

Steps to Reproduce

1. start code-server
2. use code-server as usual
3. after about one week to one month, the code-server hangs
4. we can't reproduce this issue manually as this always happens suddenly and unexpectedly

Troubleshooting so far

1. We can't open code-server in browser as chrome shows no response at all, but port access is still avaiable when we use telnet
2. We can find code-server process still exists and it is still listening on the port (with telnet 127.0.0.1 8080)
   
3. We even can't curl on the local linux machine and there's no response at all
   

(if it's working then we should see the direction infomation like)

4. We generated core file of the hang process (I mean the process executing entry.js) and use gdb to debug it. When we use bt to look at the frames, we can get the following content
   
   We get this core files from over 20+ of our users and the contents are all the same, showing nodejs is stucking at
   write() from /lib64/libpthread.so.0

Expected

nodejs server in linux backend system should response when we use curl or open IDE in browser

Actual

nodejs process in linux backend system hangs and no response at all

Logs

from ~/.local/share/code-server/code-server-stderr.log
We found the following logs and it seems that there's network traffic issue in socket connection but I don't know how we use these information to troubleshoot. I will be appreciated it if you team can suggest something.

Screenshot/Video

No response

Does this issue happen in VS Code or GitHub Codespaces?

• I cannot reproduce this in VS Code.
• I cannot reproduce this in GitHub Codespaces.

Are you accessing code-server over HTTPS?

• I am using HTTPS.

Notes

No response

[jsjoeio]

This is a tough one 🤔 Nothing obvious is coming to me after reading your logs. @code-asher any ideas? I'm not sure if long-running processes have issues or something.

What happens if you restart code-server? Does it work then?

[code-asher]

Fascinating! Yeah there was a problem with stdout not being consumed although I only saw this after a hotswap (which code-server performs when it receives a SIGUSR1 or SIGUSR2) so I am not sure how the issue would trigger without a hotswap but if it does that would explain it.

[ericzhucode]

Fascinating! Yeah there was a problem with stdout not being consumed although I only saw this after a hotswap (which code-server performs when it receives a SIGUSR1 or SIGUSR2) so I am not sure how the issue would trigger without a hotswap but if it does that would explain it.

Can I ask more about hotswap you mentioned above? I know signal SIGUSR1 and SIGUSR2 since we sometimes use that in order to do an attach debug, but what does hotswap mean? Is it a feature controlled by code-server code or by express frameworks or nodejs?

[code-asher]

In this case I mean code-server restarts itself. code-server's top-level process is a thin wrapper that starts the main code-server process and when the top-level code-server process receives SIGUSR1 or SIGUSR2 it restarts the main code-server process. I think in the pipe implementation when the main process streams closed they closed the top-level ones too.

[code-asher]

We should probably remove SIGUSR1 since that overlaps with debugging Node.

[ericzhucode]

Much appreciated for your insights!