|
1 | 1 | # Getting started with Gateway API Inference Extension
|
2 | 2 |
|
3 |
| -This quickstart guide is intended for engineers familiar with k8s and model servers (vLLM in this instance). The goal of this guide is to get a first, single InferencePool up and running! |
| 3 | +??? example "Experimental" |
| 4 | + |
| 5 | + This project is still in an alpha state and breaking changes may occur in the future. |
| 6 | + |
| 7 | +This quickstart guide is intended for engineers familiar with k8s and model servers (vLLM in this instance). The goal of this guide is to get an Inference Gateway up and running! |
4 | 8 |
|
5 | 9 | ## **Prerequisites**
|
6 | 10 | - A cluster with:
|
@@ -124,15 +128,15 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
|
124 | 128 | ./istioctl install --set tag=$TAG --set hub=gcr.io/istio-testing
|
125 | 129 | ```
|
126 | 130 |
|
127 |
| - 1. If you run the Endpoint Picker (EPP) with TLS (with `--secureServing=true`), it is currently using a self-signed certificate |
128 |
| - and the gateway cannot successfully validate the CA signature and the SAN. Apply the destination rule to bypass verification as |
129 |
| - a temporary workaround. A better TLS implementation is being discussed in [Issue 582](https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/582). |
| 131 | + 1. Deploy Gateway |
130 | 132 |
|
131 |
| - ```bash |
132 |
| - kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/destination-rule.yaml |
133 |
| - ``` |
| 133 | + ??? note |
134 | 134 |
|
135 |
| - 1. Deploy Gateway |
| 135 | + If you run the Endpoint Picker (EPP) with the `--secureServing` flag set to `true`, it is currently using a self-signed certificate. As a security measure, Istio does not trust self-signed certificates by default. As a temporary workaround, you can apply the destination rule to bypass TLS verification for EPP. A more secure TLS implementation in EPP is being discussed in [Issue 582](https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/582). |
| 136 | + |
| 137 | + ```bash |
| 138 | + kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/destination-rule.yaml |
| 139 | + ``` |
136 | 140 |
|
137 | 141 | ```bash
|
138 | 142 | kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/gateway.yaml
|
@@ -166,18 +170,14 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
|
166 | 170 | 1. Install Kgateway CRDs
|
167 | 171 |
|
168 | 172 | ```bash
|
169 |
| - helm upgrade -i --create-namespace --namespace kgateway-system --version v2.0.0-main kgateway-crds https://github.com/danehans/toolbox/raw/refs/heads/main/charts/338661f3be-kgateway-crds-1.0.1-dev.tgz |
| 173 | + helm upgrade -i --create-namespace --namespace kgateway-system --version $VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds |
170 | 174 | ```
|
171 | 175 |
|
172 | 176 | 1. Install Kgateway
|
173 | 177 |
|
174 | 178 | ```bash
|
175 |
| - helm upgrade --install kgateway "https://github.com/danehans/toolbox/raw/refs/heads/main/charts/338661f3be-kgateway-1.0.1-dev.tgz" \ |
176 |
| - -n kgateway-system \ |
177 |
| - --set image.registry=danehans \ |
178 |
| - --set image.pullPolicy=Always \ |
179 |
| - --set inferenceExtension.enabled="true" \ |
180 |
| - --version 1.0.1-dev |
| 179 | + helm upgrade -i --namespace kgateway-system --version $VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway |
| 180 | +--set inferenceExtension.enabled=true |
181 | 181 | ```
|
182 | 182 |
|
183 | 183 | 1. Deploy Gateway
|
@@ -244,7 +244,5 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
|
244 | 244 | 1. Uninstall the CRDs
|
245 | 245 |
|
246 | 246 | ```bash
|
247 |
| - kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/crd/bases/inference.networking.x-k8s.io_inferencepools.yaml --ignore-not-found |
248 |
| - kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/crd/bases/inference.networking.x-k8s.io_inferencemodels.yaml --ignore-not-found |
249 | 247 | kubectl delete -k https://github.com/kubernetes-sigs/gateway-api-inference-extension/config/crd --ignore-not-found
|
250 | 248 | ```
|
0 commit comments