Simplify POC installation #8
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
Welcome @liu-cong! |
k8s-ci-robot
added
the
size/L
| apiVersion: gateway.networking.k8s.io/v1 | ||
| kind: Gateway | ||
| apiVersion: apps/v1 | ||
| kind: Deployment |
There was a problem hiding this comment.
gatewayclass as the file name may be misleading if we are putting the ext_proc deployment yaml here
There was a problem hiding this comment.
The point is that ext_proc is an implementation detail of this specific gateway class
There was a problem hiding this comment.
A reasonable perspective.
However, the core of this PoC is the ext-proc implementation, and I don't think can be considered an implementation detail here.
There was a problem hiding this comment.
how about just call it installation.yaml which just bundles everything, or can break it down to multiple yaml files
There was a problem hiding this comment.
cool, renamed to installation.yaml. And the README explains what's being installed so this should be good!
Yes both @courageJ and I tried this. |
liu-cong
force-pushed
the
main
branch
2 times, most recently
from
fc9ce95 to
2097027
Compare
|
/lgtm |
k8s-ci-robot
added
the
lgtm
| Wait until the gateway is ready. | ||
| ```bash | ||
| IP=$(kubectl get gateway/llm-gateway -o jsonpath='{.status.addresses[0].value}') | ||
| PORT=8081 |
There was a problem hiding this comment.
Isn't this wrong, llm-instance-gw is listening on 8080. Am I wrong?
There was a problem hiding this comment.
Good question! Actually in the POC setup, Envoy is configured the additional 8081 port for ext proc traffic. Updated README
There was a problem hiding this comment.
Ho yes, you're right actually. Sorry, for the confusion.
|
Overall it looks good to me and it's a great addition. Thank you @liu-cong 😀 |
k8s-ci-robot
removed
the
lgtm
Update README on envoy 8081 port
|
/lgtm |
k8s-ci-robot
added
the
lgtm
| @@ -1,68 +1,55 @@ | |||
| # Envoy Ext Proc Gateway with LoRA Integration | |||
|
|
|||
| This project sets up an Envoy gateway to handle gRPC calls with integration of LoRA (Low-Rank Adaptation). The configuration aims to manage gRPC traffic through Envoy's external processing and custom routing based on headers and load balancing rules. The setup includes Kubernetes services and deployments for both the gRPC server and the vllm-lora application. | |||
| This project sets up an Envoy gateway with a custom external processing which implements advanced routing logic tailored for LoRA (Low-Rank Adaptation) adapters. The routing algorithm is based on the model specified (using Open AI API format), and ensuring efficient load balancing based on model server metrics. | |||
There was a problem hiding this comment.
one more extra space between which and implements
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: liu-cong, terrytangyuan The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
This was adapted from the private POC repo.
This PR separates what needs to installed vs. what resources user creates as sample applications. The installation is simplified to a single gatewayclass.yaml file.
@courageJ encountered some issue when trying the POC, and this PR worked for her. Not sure what didn't work before though :)