Terminating user pods slower than needed?

[consideRatio]

@minrk I recall you had an idea on why it took so long to kill a user pod. It was because the spawners stop command did not actually stop the notebook container but left it running?

What was needed to be done in order to fix this, so that terminating a pod can be quicker than waiting the full grace period?

  Normal  Killing                 52s   kubelet, gke-dsp-default-pool-c86f4aeb-3pxj  Killing container with id docker://notebook:Need to kill Pod

[yuvipanda]

Yeah, I've been noticing this too. I am not sure what is causing this, but it's been extremely annoying!

Possible culprits are:

1. Grace Period passed to k8s API is wrong
2. K8S is not respecting grace period properly
3. We have an intermediate bash / sh shell somewhere in the container spawn process that isn't passing signals along properly

[consideRatio]

I think @minrk will do a mic drop on 3. :D

1. is because we don't override the terminationGracePeriodSeconds of the Pod's spec, and the default is 30 seconds. It is not modified by kubespawner at all as far as saw by searching the kubespawner repo or inspecting the pod spec.

---

But we still end up with more than 30 seconds to terminate the pod, why? Perhaps 30 seconds is due to waiting on termination that isn't happening, and the rest which is about 30 additional seconds when kubernetes is attempting to shut it down and dismount the volume etc.

[consideRatio]

While we are at it, the proxy pod is also super slow to delete.

[minrk]

In my understanding, GracePeriod should ~never be relevant unless there's a problem. Shutting down a notebook server doesn't take very long (depends on how many running kernels. Typically < 1 second if no kernels), so the answer is either that the notebook server isn't getting or handling the shutdown signal as expected, or Kubernetes is spending time doing something other than shutting down the notebook process.

I'd give two likely candidates:

• shutdown is failing entirely (3. above), e.g. due to signals being captured by an ENTRYPOINT, or some other bug (e.g. jupyterhub 0.9.1 where the signal handler simply didn't trigger shutdown)
• volume unmount is slow

For instance, I stopped a pod on binder staging just now, and it was gone ~immediately (less than two seconds). So it's not a general issue with the chart, but rather something in specific deployment configurations causing the shutdown to not occur properly.

[consideRatio]

I've worked to resolve this kind of issues already, and it turned out for example that the proxy pod running CHP failed to react to a SIGTERM encouragement from the kernel, and only reacted to the SIGKILL that arrived after 30 seconds by k8s. This was btw something I could test with docker as docker stop is SIGTERM followed by SIGKILL just like in a kubectl delete results in.

See jupyterhub/configurable-http-proxy#217.