|
| 1 | +# Gateway API Inference Extension |
| 2 | + |
| 3 | +Author(s): @kfswain |
| 4 | +## Proposal Status |
| 5 | + ***Draft*** |
| 6 | + |
| 7 | +## Table of Contents |
| 8 | + |
| 9 | +<!-- toc --> |
| 10 | + |
| 11 | +- [Summary](#summary) |
| 12 | +- [Goals](#goals) |
| 13 | +- [Non-Goals](#non-goals) |
| 14 | +- [Proposal](#proposal) |
| 15 | + - [Personas](#personas) |
| 16 | + - [Inference Platform Admin](#inference-platform-admin) |
| 17 | + - [Inference Workload Owner](#workload-owner) |
| 18 | + - [Axioms](#axioms) |
| 19 | + - [InferencePool](#inferencepool) |
| 20 | + - [InferenceModel](#inferencemodel) |
| 21 | + - [Spec](#spec) |
| 22 | + - [Diagrams](#diagrams) |
| 23 | + - [Alternatives](#alternatives) |
| 24 | +- [Open Questions](#open-questions) |
| 25 | + |
| 26 | +<!-- /toc --> |
| 27 | + |
| 28 | +## Summary |
| 29 | + |
| 30 | +This proposal seeks to standardize the implementation of an EPP (End-point Picker) for the Inference Gateway extension (also known as Gateway API Inference Extension). Additionally, this proposes to restructure the current implementation of the EPP to be more modular, and approachable. |
| 31 | + |
| 32 | +## Goals |
| 33 | + |
| 34 | +- Set a standard on how the EPP & APIs interact |
| 35 | +- Settle on common nomenclature for clearer communication |
| 36 | +- Allow for modularization of the EPP, to be extended to a user's specific needs |
| 37 | + |
| 38 | +## Non-Goals |
| 39 | + |
| 40 | +- Reshaping the current API |
| 41 | +- A change in scope of the current project |
| 42 | + |
| 43 | +## Proposal |
| 44 | + |
| 45 | +This proposal is not proposing any net new features, instead, we are refactoring our current implementation to better handle more devs, more features, etc. At the time of writing, GIE is currently at v0.3, and that stronger experimental context (along with external feedback) made clear the need this restructure. The image below give a high level view of how our components work together. |
| 46 | + |
| 47 | +<img src="./images/epp_arch.svg" alt="Scheduling Algorithm" width="1000" /> |
| 48 | + |
| 49 | +## Overview |
| 50 | +At a quick glance, the EPP is being broken into specific layers. The `Data Layer` is of note, as it is a vertical that will be accessed by all the others. The data layer manages the k8s, data, metric & usage data, as well as processing of the above data to determine resource scarcity regimes. |
| 51 | + |
| 52 | +The other layers are handled in sequential process. Starting with the **Ext-Proc** call. The request is buffered and then sent to the **Routing Layer**, which processes any User defined per-InferenceModel routing rules & request enrichment happening first (at the time of writing that is currently just translating the InferenceModel name to a weight-split actual model). Then _all_ requests pass through the to-be-implemented [**Flow Controller**](https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/674) to ensure that any request entry to the pool adhereing to the guidelines set by the Priority, Fairness, & Queueing configuration. And finally, the **Scheduling Layer** is the load balancing algorithm that intelligently routes requests based on the current state of the InferencePool. |
| 53 | + |
| 54 | +## Components |
| 55 | + |
| 56 | +To further expand upon these component layers. We will first break them into `extensible` and `non-extensible` layers. `Non-extensible` layers are intended to be static, and handled on behalf of the user, typically implementing low-opinion infrastructure. |
| 57 | + |
| 58 | +The `Extensible` layers are: |
| 59 | +- Data Layer |
| 60 | +- Routing Layer |
| 61 | +- Flow Controller |
| 62 | +- Scheduling Layer |
| 63 | + |
| 64 | +The `Non-Extensible` layer(s) are: |
| 65 | +- The Ext-Proc Server |
| 66 | + |
| 67 | +### `Extensible` |
| 68 | + |
| 69 | +#### Data Layer |
| 70 | + |
| 71 | +The data layer will consume and store: the InferencePool/InferenceModel config and the pre-defined [Model Server Protocol](../003-model-server-protocol/README.md). Additionally, the data fed from the model servers will be processed and digested to provide resource scarcity regime hints, and autoscaling reccomendations. |
| 72 | + |
| 73 | +Many extensions to scheduling will require changes to ingested metrics, as such, the data layer will be built to be extended, but extenders accept that the Model Server Protocol will no longer provide guarantees on portability of a model server out of the box. |
| 74 | + |
| 75 | +#### Routing Layer |
| 76 | + |
| 77 | +The routing layer is likely to be the most opinion heavy section, as the scope of what constitutes a 'Route Rule' is somewhat broad. The current examples we expect would be: |
| 78 | + |
| 79 | +- System Prompt injection |
| 80 | +- RAG callout |
| 81 | +- Per-InferenceModel request validation (such as saftey/on-topic, etc) |
| 82 | + |
| 83 | +Due to the possibility of this becoming a bit of a dumping ground. The API will keep a _very_ tight scope on which of these route rules are included in the spec. A standard method of extension will be provided if the need to define a custom rule arises. |
| 84 | + |
| 85 | +#### Flow Controller (WIP - implementation tracked in [#674](https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/674)) |
| 86 | + |
| 87 | +The flow controller will consume resource regime data, and enforce proper resource sharing between workloads. This will primarily be done through a queuing mechanism [as described here](https://docs.google.com/document/d/1VZL7opFWuwgWquvgiOzLlXAJ633qZ9U-A0ZixGjBgaI/edit?usp=sharing). |
| 88 | + |
| 89 | +#### Scheduling Layer |
| 90 | + |
| 91 | +As the Scheduling Layer is the final interface to the entirety of the pool, all configuration will be at the _pool_ level. The default scheduling layer will be an experimentally-backed LB algorithm, with exposed config values. |
| 92 | + |
| 93 | +The Scheduler will define a strong interface API, so that new scheduling algos may be plugged & dark-launched to test in production traffic without impacting said traffic. Extension is expected to adhere to the [Scheduler Subsystem definition](https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/603) |
| 94 | + |
| 95 | +### `Non-extensible` |
| 96 | + |
| 97 | +#### Ext-Proc Server |
| 98 | + |
| 99 | +The Ext-Proc Server protocol is very well defined & specific, deviation could cause the EPP to become unusable or unstable. Extension is ill-advised. |
0 commit comments