Introduce keep alive and fix AWS Load Balancer 502 errors #491

a3626a · 2023-08-17T06:14:02Z

I'm running Z2JH based service with about 1,000 DAU. It is deployed in AWS EKS attached to AWS ALB.

As DAU grows, users started to get 502 Responses from the LB.

This is well-known problem related to keep-alive setting. (AWS Article)
Unfortunately, configurable-http-proxy does not support keep-alive. So I implemented, and tested in production environment.

After the deployment the number of 502 errors descreased.

Technical/Implementation detail

It is very important to allow keep-alive both client side and server side. That's why Agent and keepAliveTimeout are both needed.
The jupyter hub and jupyter server support keep-alive by default, because they are Tornado servers.
chp is given these parameters. They are AWS specific values.

"--server-keep-alive-timeout=61000"
"--agent-free-socket-timeout=62000"

welcome · 2023-08-17T06:14:04Z

Thanks for submitting your first pull request! You are awesome! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly.

You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

minrk · 2023-08-17T07:32:20Z

Thanks! I think enabling keep-alive makes sense, and exposing the timeout as an option is sensible as well. What I'm trying to understand is the addition of the keepaliveagent package instead of using the standard-library server.keepAliveTimeout. Can you speak to that as to why it's needed beyond setting keepAlive: true, keepAliveTimeout: 62000?

minrk · 2023-08-17T07:37:45Z

#481 enables keep-alive, which I think makes sense, and combining it with this to expose the keep-alive timeout as an option seems like the right way to go, unless the keepaliveagent solves a problem I'm not quite seeing.

a3626a · 2023-08-17T08:54:56Z

I have done some experiments and concluded that keep alive should be supported for the both directions (client side - Load Balancer, and server side - Jupyter Hub or Jupyter Server)

But my experiment was not well organized to be shared. I haved used curl to check keep-alive support, and nc(Netcat) to verify timeout. I have found that without agent keep-alive connections are closed after 5 seconds, even though I have set server.keepAliveTimeout to 60 seconds.

I will do the experiment again, and share it here.

a3626a · 2023-08-17T09:28:59Z

I have done simple experiment again.

I opened a shell inside the proxy pod which is deployed by Z2JH. Then I executed curl -v localhost:8000.

CASE 1) `--server-keep-alive-timeout=15000` && `--agent-free-socket-timeout=16000`

Check the arguments using ps

/srv/configurable-http-proxy $ ps | grep node
    1 nobody    0:03 node /srv/configurable-http-proxy/bin/configurable-http-proxy --ip= --api-ip= --api-port=8001 --default-target=http://jupyterhub1-hub:8081 --error-target=http://jupyterhub1-hub:8081/hub/error --port=8000 --log-level=debug --metrics-port=8080 --server-keep-alive-timeout=15000 --agent-free-socket-timeout=16000

/srv/configurable-http-proxy $ curl -v localhost:8000
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 302 Found
< server: TornadoServer/6.2
< content-type: text/html
< date: Thu, 17 Aug 2023 09:24:46 GMT
< access-control-allow-origin: *
< access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS
< content-security-policy: frame-ancestors self codle.io dev.codle.io
< x-jupyterhub-version: 3.0.0
< access-control-allow-headers: accept, content-type, authorization
< location: /hub/
< content-length: 0
< connection: keep-alive
< 
* Connection #0 to host localhost left intact

-> left intact means keep-alive works.

CASE 2) `--server-keep-alive-timeout=15000`

Check the arguments, too.

/srv/configurable-http-proxy $ ps | grep node
    1 nobody    0:04 node /srv/configurable-http-proxy/bin/configurable-http-proxy --ip= --api-ip= --api-port=8001 --default-target=http://jupyterhub2-hub:8081 --error-target=http://jupyterhub2-hub:8081/hub/error --port=8000 --log-level=debug --metrics-port=8080 --server-keep-alive-timeout=15000

/srv/configurable-http-proxy $ curl -v localhost:8000
*   Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 302 Found
< server: TornadoServer/6.2
< content-type: text/html
< date: Thu, 17 Aug 2023 09:24:38 GMT
< access-control-allow-origin: *
< access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS
< content-security-policy: frame-ancestors self codle.io dev.codle.io
< x-jupyterhub-version: 3.0.0
< access-control-allow-headers: accept, content-type, authorization
< location: /hub/
< content-length: 0
< connection: close
< 
* Closing connection 0

No keep-alive.

minrk · 2023-08-17T13:58:41Z

Can you test with #492? It seems to enable keep-alive all the way through from proxied requests from tornado.

minrk · 2023-08-17T14:12:15Z

Actually, there seems to be something weird where we can't use a single agent for keep-alive on both http or https with the standard library (bizarre), so I think maybe this PR is the way to go.

a3626a · 2023-08-18T01:22:17Z

Actually, there seems to be something weird where we can't use a single agent for keep-alive on both http or https with the standard library (bizarre), so I think maybe this PR is the way to go.

For the agentkeepalive library, I followed this example. But there're no particular reason or cases that this library must be used. http.Agent could work, I am not sure.

Can you test with #492? It seems to enable keep-alive all the way through from proxied requests from tornado.

Ok. I will post curl result and also nc result. I think #492 will do keep-alive just for 5 seconds, won't respect the given timeout argument. Because timeout is not passed to the agent.

Also, I set up TLS termination on LB, so all my tests are done using HTTP.

a3626a · 2023-08-18T02:31:36Z

Fix

#492 has issue.

01:45:21.756 [ConfigProxy] info: Adding route / -> http://jupyterhub2-hub:8081
node:internal/validators:96
      throw new ERR_INVALID_ARG_TYPE(name, 'number', value);
      ^

TypeError [ERR_INVALID_ARG_TYPE]: The "keepAliveTimeout" argument must be of type number. Received type string ('15000')
    at Server.storeHTTPOptions (node:_http_server:464:5)
    at new Server (node:_http_server:507:20)
    at Object.createServer (node:http:61:10)
    at new ConfigurableProxy (/srv/configurable-http-proxy/lib/configproxy.js:234:31)
    at Object.<anonymous> (/srv/configurable-http-proxy/bin/configurable-http-proxy:320:13)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {
  code: 'ERR_INVALID_ARG_TYPE'
}

Node.js v18.16.0
Stream closed EOF for jupyter-hub/jupyterhub2-proxy-58b65b6b87-drnfn (chp)

I added parseInt and test again. (I substituted 5000 to parseInt)

Curl Test

ps result

/srv/configurable-http-proxy $ ps | grep node
    1 nobody    0:01 node /srv/configurable-http-proxy/bin/configurable-http-proxy --ip= --api-ip= --api-port=8001 --default-target=http://jupyterhub2-hub:8081 --error-target=http://jupyterhub2-hub:8081/hub/error --port=8000 --log-level=debug --metrics-port=8080 --keep-alive-timeout=15000

/srv/configurable-http-proxy $ curl -v localhost:8000
* processing: localhost:8000
*   Trying [::1]:8000...
* Connected to localhost (::1) port 8000
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.2.1
> Accept: */*
> 
< HTTP/1.1 302 Found
< server: TornadoServer/6.2
< content-type: text/html
< date: Fri, 18 Aug 2023 02:20:21 GMT
< access-control-allow-origin: *
< access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS
< content-security-policy: frame-ancestors self codle.io dev.codle.io
< x-jupyterhub-version: 3.0.0
< access-control-allow-headers: accept, content-type, authorization
< location: /hub/
< content-length: 0
< connection: keep-alive
< 
* Connection #0 to host localhost left intact

Keep alive works.

Netcat Test

nc test is done manully, very naive.

CASE 1 : Timeout=15000, Request, Wait 10 seconds, Request again

ps result

/srv/configurable-http-proxy $ ps | grep node
    1 nobody    0:01 node /srv/configurable-http-proxy/bin/configurable-http-proxy --ip= --api-ip= --api-port=8001 --default-target=http://jupyterhub2-hub:8081 --error-target=http://jupyterhub2-hub:8081/hub/error --port=8000 --log-level=debug --metrics-port=8080 --keep-alive-timeout=15000

/srv/configurable-http-proxy $ nc localhost 8000
GET / HTTP/1.1

HTTP/1.1 302 Found
server: TornadoServer/6.2
content-type: text/html
date: Fri, 18 Aug 2023 02:24:23 GMT
access-control-allow-origin: *
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS
content-security-policy: frame-ancestors self codle.io dev.codle.io
x-jupyterhub-version: 3.0.0
access-control-allow-headers: accept, content-type, authorization
location: /hub/
content-length: 0
connection: keep-alive

< Wait 10 Seconds >

GET / HTTP/1.1

HTTP/1.1 302 Found
server: TornadoServer/6.2
content-type: text/html
date: Fri, 18 Aug 2023 02:24:34 GMT
access-control-allow-origin: *
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS
content-security-policy: frame-ancestors self codle.io dev.codle.io
x-jupyterhub-version: 3.0.0
access-control-allow-headers: accept, content-type, authorization
location: /hub/
content-length: 0
connection: keep-alive

It should keep alive after 10 seconds, it actually does.

CASE 2 : Timeout=15000, Request, Wait 20 seconds, Request again

ps result

/srv/configurable-http-proxy $ ps | grep node
    1 nobody    0:03 node /srv/configurable-http-proxy/bin/configurable-http-proxy --ip= --api-ip= --api-port=8001 --default-target=http://jupyterhub2-hub:8081 --error-target=http://jupyterhub2-hub:8081/hub/error --port=8000 --log-level=debug --metrics-port=8080 --keep-alive-timeout=15000

/srv/configurable-http-proxy $ nc localhost 8000
GET / HTTP/1.1

HTTP/1.1 302 Found
server: TornadoServer/6.2
content-type: text/html
date: Fri, 18 Aug 2023 02:25:55 GMT
access-control-allow-origin: *
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS
content-security-policy: frame-ancestors self codle.io dev.codle.io
x-jupyterhub-version: 3.0.0
access-control-allow-headers: accept, content-type, authorization
location: /hub/
content-length: 0
connection: keep-alive

< Wait 20 Seconds >

GET / HTTP/1.1

< Connection Closed >

It should close connection after 20 seconds, it actually does.

Conclusion

I think #492 works with a parseInt fix.

I thought #492 would not respect the given timeout. However, standard library http.Agent seems like closing connections when the number of connections exceeds its limit. It does not close idle connections. So it works like infinite timeout when the number of active connection is below the limit.

Add agent and keepAlive

fafd40f

Install missing dependencies

b0a6b83

a3626a changed the title ~~Introduce keep alive and fix AWS Load Balancer 502~~ Introduce keep alive and fix AWS Load Balancer 502 errors Aug 17, 2023

Improve argument comment

46916d1

minrk mentioned this pull request Aug 17, 2023

Enable keep-alive, --keep-alive-timeout flag added (default: 5000ms) #492

Merged

consideRatio closed this in #492 Aug 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce keep alive and fix AWS Load Balancer 502 errors #491

Introduce keep alive and fix AWS Load Balancer 502 errors #491

a3626a commented Aug 17, 2023 •

edited

Loading

welcome bot commented Aug 17, 2023

minrk commented Aug 17, 2023 •

edited

Loading

minrk commented Aug 17, 2023

a3626a commented Aug 17, 2023

a3626a commented Aug 17, 2023 •

edited

Loading

minrk commented Aug 17, 2023

minrk commented Aug 17, 2023

a3626a commented Aug 18, 2023 •

edited

Loading

a3626a commented Aug 18, 2023 •

edited

Loading

Introduce keep alive and fix AWS Load Balancer 502 errors #491

Introduce keep alive and fix AWS Load Balancer 502 errors #491

Conversation

a3626a commented Aug 17, 2023 • edited Loading

welcome bot commented Aug 17, 2023

minrk commented Aug 17, 2023 • edited Loading

minrk commented Aug 17, 2023

a3626a commented Aug 17, 2023

a3626a commented Aug 17, 2023 • edited Loading

CASE 1) --server-keep-alive-timeout=15000 && --agent-free-socket-timeout=16000

CASE 2) --server-keep-alive-timeout=15000

minrk commented Aug 17, 2023

minrk commented Aug 17, 2023

a3626a commented Aug 18, 2023 • edited Loading

a3626a commented Aug 18, 2023 • edited Loading

Fix

Curl Test

Netcat Test

Conclusion

a3626a commented Aug 17, 2023 •

edited

Loading

minrk commented Aug 17, 2023 •

edited

Loading

a3626a commented Aug 17, 2023 •

edited

Loading

CASE 1) `--server-keep-alive-timeout=15000` && `--agent-free-socket-timeout=16000`

CASE 2) `--server-keep-alive-timeout=15000`

a3626a commented Aug 18, 2023 •

edited

Loading

a3626a commented Aug 18, 2023 •

edited

Loading