InferencePool: Add BBR Config

[danehans]

Currently, inferencepool.spec inlines EndpointPickerConfig that is defined by the following:

Specifies the configuration needed by the proxy to discover and connect to the endpoint picker service that picks endpoints for the requests routed to this pool.

The Body Based Routing (BBR) extension was recently added, but InferencePool was not updated. Does BBR act independently of an InferencePool? If so, how does a proxy discover and connect with this extension?

[kfswain]

Yeah, it should act independently of the InferencePool (as it's not strictly necessary, and useful more for routing via HTTPRoute rules) @rramkumar1 I believe added some documentation for how this should/could be used in OSS. If not we absolutely should have something

[kfswain]

Perhaps this convo would tie into: kubernetes-sigs/gateway-api#3625 (comment) (except somewhat different, as it would happen before the HTTPRoute)

[rramkumar1]

@danehans This won't need any config in the InferencePool, it's deployed separately.

Below is an example of how it would have been deployed using Istio GW (note InferencePool support there is still experimental).

https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/config/charts/body-based-routing/templates/istio.yaml

Other LB implementations would do something similar depending on how they allow ext_proc configuration.

[nirrozenbaum]

@rramkumar1 I went over this chart. overall looks good and very clear.
I just don't see there anything related to EPP which makes sense according to the this response.

it's deployed separately.

do have somewhere a documentation of the full flow using BBR + EPP?
how would one connect BBR to EPP if he wishes to do so?

[rramkumar1]

Sure I'll take an AI to add some more docs on this.

/assign