Configure the vllm deployment with best practices for startup

We want to recommend best practices for deployments of model servers
under an InferencePool. Use the need to gracefully drain without
client visible errors during rollout ("hitless" updates) to
annotate the yaml with strong opinions on best practices.

This configuration was experimentally verified on the GKE Inference
Gateway configuration which should be longer than other servers.

Author: smarterclayton

config/manifests/vllm/gpu-deployment.yaml

@@ -46,26 +46,111 @@ spec:
             - containerPort: 8000
               name: http
               protocol: TCP
+          lifecycle:
+            preStop:
+              # vLLM stops accepting connections when it receives SIGTERM, so we need to sleep
+              # to give upstream gateways a chance to take us out of rotation. The time we wait
+              # is dependent on the time it takes for all upstreams to completely remove us from
+              # rotation. Older or simpler load balancers might take upwards of 30s, but we expect
+              # our deployment to run behind a modern gateway like Envoy which is designed to 
+              # probe for readiness aggressively.
+              sleep:
+                # Upstream gateway probers for health should be set on a low period, such as 5s,
+                # and the shorter we can tighten that bound the faster that we release
+                # accelerators during controlled shutdowns. However, we should expect variance,
+                # as load balancers may have internal delays, and we don't want to drop requests
+                # normally, so we're often aiming to set this value to a p99 propagation latency
+                # of readiness -> load balancer taking backend out of rotation, not the average.
+                # 
+                # This value is generally stable and must often be experimentally determined on
+                # for a given load balancer and health check period. We set the value here to
+                # the highest value we observe on a supported load balancer, and we recommend
+                # tuning this value down and verifying no requests are dropped.
+                #
+                # If this value is updated, be sure to update terminationGracePeriodSeconds.
+                #
+                seconds: 30
+              #
+              # IMPORTANT: preStop.sleep is beta as of Kubernetes 1.30 - for older versions
+              # replace with this exec action.
+              #exec:
+              #  command:
+              #  - /usr/bin/sleep
+              #  - 30
           livenessProbe:
-            failureThreshold: 240
             httpGet:
               path: /health
               port: http
               scheme: HTTP
-            initialDelaySeconds: 5
-            periodSeconds: 5
+            # vLLM's health check is simple, so we can more aggressively probe it.  Liveness
+            # check endpoints should always be suitable for aggressive probing.
+            periodSeconds: 1
             successThreshold: 1
+            # vLLM has a very simple health implementation, which means that any failure is
+            # likely significant. However, any liveness triggered restart requires the very
+            # large core model to be reloaded, and so we should bias towards ensuring the
+            # server is definitely unhealthy vs immediately restarting. Use 5 attempts as
+            # evidence of a serious problem.
+            failureThreshold: 5
             timeoutSeconds: 1
           readinessProbe:
-            failureThreshold: 600
             httpGet:
               path: /health
               port: http
               scheme: HTTP
-            initialDelaySeconds: 5
-            periodSeconds: 5
+            # vLLM's health check is simple, so we can more aggressively probe it.  Readiness
+            # check endpoints should always be suitable for aggressive probing, but may be
+            # slightly more expensive than readiness probes.
+            periodSeconds: 1
             successThreshold: 1
+            # vLLM has a very simple health implementation, which means that any failure is
+            # likely significant,
+            failureThreshold: 1
             timeoutSeconds: 1
+          # We set a startup probe so that we don't begin directing traffic to this instance
+          # until the model is loaded.
+          startupProbe:
+            # Failure threshold is when we believe startup will not happen at all, and is set
+            # to the maximum possible time we believe loading a model will take. In our
+            # default configuration we are downloading a model from HuggingFace, which may
+            # take a long time, then the model must load into the accelerator. We choose
+            # 10 minutes as a reasonable maximum startup time before giving up and attempting
+            # to restart the pod.
+            #
+            # IMPORTANT: If the core model takes more than 10 minutes to load, pods will crash
+            # loop forever. Be sure to set this appropriately.
+            failureThreshold: 600
+            # Set delay to start low so that if the base model changes to something smaller
+            # or an optimization is deployed, we don't wait unneccesarily.
+            initialDelaySeconds: 2
+            # As a startup probe, this stops running and so we can more aggressively probe
+            # even a moderately complex startup - this is a very important workload.
+            periodSeconds: 1
+            exec:
+              # Verify that our core model is loaded before we consider startup successful.
+              # /health starts returning true very early in vLLM startup, but we want to
+              # only consider ourselves as started up once the model has been loaded.
+              #
+              # vLLM should implement a readiness check that is only true once the model
+              # can begin serving, and then this can be switched to an httpGet probe.
+              # https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/558
+              command:
+              - /bin/bash
+              - -c
+              - |
+                set -eu
+                if ! models="$( curl -q http://0.0.0.0:8000/v1/models )"; then
+                  echo "server not responding"
+                  exit 1
+                fi
+                echo "${models}" | grep -q "$1"
+                if [[ $? -ne 0 ]]; then
+                  echo "model not found"
+                  exit 1
+                fi
+                echo "ok"
+              - ''
+              - '"id":"meta-llama/Llama-2-7b-hf"'
           resources:
             limits:
               nvidia.com/gpu: 1
@@ -92,8 +177,65 @@ spec:
           - name: config-volume
             mountPath:  /config
       restartPolicy: Always
-      schedulerName: default-scheduler
-      terminationGracePeriodSeconds: 30
+
+      # Generally, the termination grace period needs to last longer than the slowest request
+      # we expect to serve plus any extra time spent waiting for load balancers to take the
+      # model server out of rotation.
+      #
+      # An easy starting point is the p99 or max request latency measured for your workload,
+      # although LLM request latencies vary significantly if clients send longer inputs or
+      # trigger longer outputs. Since steady state p99 will be higher than the latency
+      # to drain a server, you may wish to slightly this value either experimentally or
+      # via the calculation below.
+      #
+      # For most models you can derive an upper bound for the maximum drain latency as
+      # follows:
+      # 
+      #   1. Identify the maximum context length the model was trained on, or the maximum
+      #      allowed length of output tokens configured on vLLM (llama2-7b was trained to
+      #      4k context length, while llama3-8b was trained to 128k).
+      #   2. Output tokens are the more compute intensive to calculate and the accelerator
+      #      will have a maximum concurrency (batch size) - the time per output token at
+      #      maximum batch with no prompt tokens being processed is the slowest an output
+      #      token can be generated (for this model it would be about 100ms TPOT at a max
+      #      batch size around 50)
+      #   3. Calculate the worst case request duration if a request starts immediately
+      #      before the server stops accepting new connections - generally when it receives
+      #      SIGTERM (for this model that is about 4096 / 10 ~ 40s)
+      #   4. If there are any requests generating prompt tokens that will delay when those
+      #      output tokens start, and prompt token generation is roughly 6x faster than
+      #      compute-bound output token generation, so add 20% to the time from above (40s + 
+      #      16s ~ 55s)
+      #
+      # Thus we think it will take us at worst about 55s to complete the longest possible
+      # request the model is likely to receive at maximum concurrency (highest latency)
+      # once requests stop being sent.
+      #
+      # NOTE: This number will be lower than steady state p99 latency since we stop receiving
+      #       new requests which require continuous prompt token computation.
+      # NOTE: The max timeout for backend connections from gateway to model servers should
+      #       be configured based on steady state p99 latency, not drain p99 latency
+      #
+      #   5. Add the time the pod takes in its preStop hook to allow the load balancers have
+      #      stopped sending us new requests (55s + 30s ~ 85s)
+      #
+      # Because termination grace period controls when the Kubelet forcibly terminates a
+      # stuck or hung process (a possibility due to a GPU crash), there is operational safety
+      # in keeping the value roughly proportional to the time to finish serving. There is also
+      # value in adding a bit of extra time to deal with unexpectedly long workloads.
+      #   
+      #   6. Add a 50% safety buffer to this time since the operational impact should be low
+      #      (85s * 1.5 ~ 130s)
+      #
+      # One additional source of drain latency is that some workloads may run close to
+      # saturation and have queued requests on each server. Since traffic in excess of the
+      # max sustainable QPS will result in timeouts as the queues grow, we assume that failure
+      # to drain in time due to excess queues at the time of shutdown is an expected failure
+      # mode of server overload. If your workload occasionally experiences high queue depths
+      # due to periodic traffic, consider increasing the safety margin above to account for
+      # time to drain queued requests.
+      terminationGracePeriodSeconds: 130
+
       volumes:
         - name: data
           emptyDir: {}