[WIP] proposal - add readypods field to InferencePoolStatus

[nirrozenbaum]

This PR suggests adding to InferencePool status field ReadyPods to reflect the number of pods that are associated with the pool and are ready to serve inference requests.
A pod is considered ready if its Ready condition is set to "True" and it is not marked for deletion.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nirrozenbaum
Once this PR has been reviewed and has the lgtm label, please assign kfswain for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	6b2fc2f
🔍 Latest deploy log	https://app.netlify.com/sites/gateway-api-inference-extension/deploys/680407e209556e0008e4ae1c
😎 Deploy Preview	https://deploy-preview-714--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[nirrozenbaum]

cc: @ahg-g @kfswain @danehans @shaneutt
let me know if this makes sense to you.

if this proposal is accepted, I'll add code to update this field in the reconcilers (should be straight forward).

[ahg-g]

We discussed this previously, see #342

@robscott for thoughts as well.

[robscott]

What is the goal for this field? Historically in Kubernetes we've tried to ensure a separation of concerns here. Ideally the portion of a Gateway implementation that handles endpoint readiness should be something that can be disconnected from higher level routing config.

More broadly, I'd be worried that trying to keep track of this in status would dramatically increase the number of updates to InferencePool status and the overall load on the controller. Instead of needing to reconcile when an InferencePool changed, it would need to reconcile anytime the readiness or number of connected endpoints changed.

[danehans]

+1 to ^. I prefer continuing to use the ResolvedRefs=False condition to indicate the following:

1. spec.selector returns 0 Pods.
2. ExtensionReference can be resolved.

@robscott @kfswain @ahg-g is my understanding of ResolvedRefs consistent with yours?

[nirrozenbaum]

More broadly, I'd be worried that trying to keep track of this in status would dramatically increase the number of updates to InferencePool status and the overall load on the controller. Instead of needing to reconcile when an InferencePool changed, it would need to reconcile anytime the readiness or number of connected endpoints changed.

@robscott that's a fair point. GIE response time to inference requests is very sensitive and this change may increase the number of reconcile events dramatically as you explained.

my thinking was to allow an easy and native way for visibility of the high level state of the pool. but when considering the tradeoff, it's not worth it. visibility can be seen also in prometheus.

closing the issue.