Add hermetic tests on the routing algorithm

[liu-cong]

Hermetic test is a fast way to verify functionality e2e without requiring an integration test env. In addition to the basic test we have today, we should add the following:

Test framework improvements:

• Run a fake k8s API server so we don't need to fake the reconcilers, done in Refactor ext-proc Main with Server Package Add Hermetic Test with k8s API Client for EPP #222

Test Cases

• Happy path: Configure one InferencePool and InferenceModel, and verify request success
• Test algorithm
  • Verify min KV cache algo without LoRA - EPP should pick the pod with least kv-cache, refer to the filter_test.go for the conditions of the pod metrics.
  • Verify LoRA affinity algo - EPP should pick the pod with LoRA already active.
  • Sheddable request succeeds when resource is available; and dropped when resource is constrained.

### Tasks
- [ ] https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/153

[ahg-g]

Please take a look at integration test setup in jobset as an example: https://github.com/kubernetes-sigs/jobset/tree/main/test/integration/controller

[danehans]

I may not be able to add every requested test case but I can get this effort started.

/assign

[liu-cong]

@danehans thanks! Please sync with @BenjaminBraunDev as he started with some initial work. There are lot of tests to add so you can divide and conquer!

[danehans]

@liu-cong thanks for the heads-up. @BenjaminBraunDev can you provide a status update when you have a moment? Do you have a local branch or WIP PR that can be referenced to see how we can divide and conquer this issue?

[danehans]

Unassigning myself why I work on the initial e2e tests.

/unassign

[BenjaminBraunDev]

@danehans Yes, I'm currently working on specifically getting a k8s Client working with the tests for InferencePool and InferenceModel CRDs. Instead of using the StartExtProc() function in test utils, I'm moving a lot of the functionality from main.go into a new package "server" and calling it with specific vars in main.go as well as in the hermetic tests. This allows the test to pipe its own environment config into the functionality of main which more accurately reflects en e2e test, and from there the hermetic test creates a k8s client using the same environment.

But in short yes, I'm working on the hermetic tests at the moment, but no I don't have a PR yet.

[danehans]

I'm moving a lot of the functionality from main.go into a new package "server" and calling it with specific vars in main.go as well as in the hermetic tests.

@BenjaminBraunDev note that #148 refactors main().

[ahg-g]

@BenjaminBraunDev how many of the tests listed are covered? can we close this?

[liu-cong]

@BenjaminBraunDev how many of the tests listed are covered? can we close this?

I updated the list. Two major items left: test error cases, verify metrics. We covered the most important test cases, which is covering the different algorithm aspsects. I propose we repurpose this to "integrate test happy paths" and close it.

I can open new issues for the remaining, so we can get more contributors to work on these.

Thoughts? @ahg-g @danehans

[ahg-g]

yeah, I think we should open separate issues for the specific tests we want to add.