It should be possible to use InferencePool without ext_proc

[howardjohn]

What would you like to be added:
The InferencePool current hardcodes that a user MUST specify and Endpoint picker.

gateway-api-inference-extension/api/v1alpha2/inferencepool_types.go

Line 69 in 6058b09

EndpointPickerConfig `json:",inline"`

.

As a vendor agnostic API, we should instead let the API drive behavior not implementation. An implementation should be free to chose it's own implementation of the semantics of InferencePool

[ahg-g]

Agreed, the idea behind the EndpointPickerConfig is to provide a one-of config, currently the only option is ExtensionRef, and we were planning to relax the required constraint on it to allow other choices. The ExtensionRef path is needed to allow for bring-your-own EPP.

Do you have something concrete in mind to spec as a second option to ExtensionRef? Or you want to allow not setting a configuration at all and leave it completely to the provider? The latter is also possible.

[howardjohn]

I think have an "implementation specific" option that is just up to the provider is probably a good start. I'm not sure what else would be standardized

[shaneutt]

Agreed. 👍 The API needs to be more generally applicable, and not coupled so tightly with a specific implementation. Some kind of resolution here would be ideal as a requirement for a GA release.

[mlavacca]

+1 Having the ext_proc requirement in the API makes this API not vendor-agnostic, as many implementations don't currently support this protocol.

[ahg-g]

Note that the current API is designed with that in mind as I mentioned above, the EndpointPickerConfig was proposed as a union of options: we can add another option to EndpointPickerConfig and relax the "required" constraint on ExtensionRef, both are non-breaking changes that we can do at any point.

If there is agreement on the above, the next step is to propose a second option within EndpointPickerConfig, perhaps add an enum to it and make it officially a union?

[hzxuzhonghu]

+1 to optional

[kfswain]

Agreed