Add metrics on metric probing

[liu-cong]

• Latency: Low metric fetch latency is crucial as the load balancing algorithm relies on near real time metrics from the model servers. Therefore it's important to monitor the metric fetch latency.
  Currently we simply log the latency, we should integrate Prometheus and add a Histogram metric.
• Error count. This can help detect bad backends.

Other consideration: We may need to sample the metrics instead of recording every probing call, due to the high probing frequency.

[MadhavJivrajani]

I'd like to try this out
/assign

[kfswain]

@JeffLuoo just an FYI

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale