Can we configure multiple inference pools by reconciling InferencePool?

[Kuromesi]

Hello guys, I'm interested in this project and wondering why the inference pool is specified when starting up and only one pool can be defined? Is it intentionally or just convenient for poc? Can we configure multiple inference pools by reconciling InferencePool resource?

[danehans]

I'm new to the project, but I suspect the design approach focuses on a single use case that is supported by one InferencePool.

[Kuromesi]

I'm new to the project, but I suspect the design approach focuses on a single use case that is supported by one InferencePool.

Thanks! I agree with that, maybe I could do some work to make some improvements on this so that we can define more pools and make it more flexible.

[ahg-g]

The APIs don't enforce a 1:1 extension to InferencePool relationship. The current reference implementation of the extension does make that assumption however, but this is more to start simple and also for sharding and scalability.

I don't believe that the ext-proc has to fundamentally be 1:1, so please feel free to make a proposal to change the code to support more than one InfernecePool. The key is to ensure that it scales well.

[Kuromesi]

The APIs don't enforce a 1:1 extension to InferencePool relationship. The current reference implementation of the extension does make that assumption however, but this is more to start simple and also for sharding and scalability.

I don't believe that the ext-proc has to fundamentally be 1:1, so please feel free to make a proposal to change the code to support more than one InfernecePool. The key is to ensure that it scales well.

Got it, thanks! :)

[danehans]

The APIs don't enforce a 1:1 extension to InferencePool relationship.

@ahg-g From my understanding, this is no longer the case since the EndpointPickerConfig type was added. This could change in the future when other extension types are added.