Manual Scale Test for Data and Control Plane Separation

[mpstefan]

As a maintainer of NGF
I want to ensure that when we scale to ~1000 Agent connections to our control plane
So that we can ensure our control plane does not get overwhelmed with connections when deployed with a highly scale data plane.

Acceptance

• Manual tests needed:
  • Test scaling NGF to 1000 Gateways to ensure the control plane is not overwhelmed by Agent connections.
  • Test if our control plane is scaled to 2 or more pods, when the 1000 data plane pods switch leaders that the control plane is not overwhelmed.
• Record breaking points if applicable.

[salonichf5]

Unassigning myself from this story until we figure out the rollback restart issue of pods during leader election.

[salonichf5]

I have collected the results in this report.

General overview - with one control plane, I am able to scale to 700 nginx instances without crashes. I tried scaling to 1000 pods with 20 control plane instances and all NGINX pods terminated. I saw only 913 pods become running before they all crashed.

Some of the error messages that seemed concerning with regards to lease acquiring deadlocks have been noted in the report.

Manual Scale Tests for Control Plane and Data Plane.docx

[sjberman]

Found some issues with Locks in the code that I am fixing that should help with scaling.

[sindhushiv]

@sjberman, Do we need to do a scale test again after the fixes are done?

[sjberman]

@sindhushiv, already did this manually. Results look much better. No issues seen.

[salonichf5]

@sjberman @sindhushiv let me know if you need any help from my side