You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pkg/README.md
+19-12
Original file line number
Diff line number
Diff line change
@@ -4,31 +4,34 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
4
4
5
5
### Requirements
6
6
- Envoy Gateway [v1.2.1](https://gateway.envoyproxy.io/docs/install/install-yaml/#install-with-yaml) or higher
7
-
- A cluster that has built-in support for `ServiceType=LoadBalancer`. (This can be validated by ensuring your Envoy Gateway is up and running)
8
-
- For example, with Kind, you can follow these steps: https://kind.sigs.k8s.io/docs/user/loadbalancer
7
+
- A cluster with:
8
+
- Support for Services of type `LoadBalancer`. (This can be validated by ensuring your Envoy Gateway is up and running). For example, with Kind,
9
+
you can follow [these steps](https://kind.sigs.k8s.io/docs/user/loadbalancer).
10
+
- 3 GPUs to run the sample model server. Adjust the number of replicas in `./manifests/vllm/deployment.yaml` as needed.
9
11
10
12
### Steps
11
13
12
-
1.**Deploy Sample vLLM Application**
14
+
1.**Deploy Sample Model Server**
13
15
14
-
Create a Hugging Face secret to download the model [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf). Ensure that the token grants access to this model.
16
+
Create a Hugging Face secret to download the model [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf). Ensure that the token grants access to this model.
15
17
Deploy a sample vLLM deployment with the proper protocol to work with the LLM Instance Gateway.
16
18
```bash
17
19
kubectl create secret generic hf-token --from-literal=token=$HF_TOKEN# Your Hugging Face Token with access to Llama2
> **_NOTE:_** This file couples together the gateway infra and the HTTPRoute infra for a convenient, quick startup. Creating additional/different InferencePools on the same gateway will require an additional set of: `Backend`, `HTTPRoute`, the resources included in the `./manifests/gateway/ext-proc.yaml` file, and an additional `./manifests/gateway/patch_policy.yaml` file. ***Should you choose to experiment, familiarity with xDS and Envoy are very useful.***
49
-
50
-
51
52
53
+
Confirm that the Gateway was assigned an IP address and reports a `Programmed=True` status:
This document provides instructions on how to run the end-to-end tests.
4
+
5
+
## Overview
6
+
7
+
The end-to-end tests are designed to validate end-to-end Gateway API Inference Extension functionality. These tests are executed against a Kubernetes cluster and use the Ginkgo testing framework to ensure the extension behaves as expected.
8
+
9
+
## Prerequisites
10
+
11
+
-[Go](https://golang.org/doc/install) installed on your machine.
12
+
-[Make](https://www.gnu.org/software/make/manual/make.html) installed to run the end-to-end test target.
13
+
- A Hugging Face Hub token with access to the [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) model.
14
+
15
+
## Running the End-to-End Tests
16
+
17
+
Follow these steps to run the end-to-end tests:
18
+
19
+
1.**Clone the Repository**: Clone the `gateway-api-inference-extension` repository:
0 commit comments