Performance test PVC with a single node saturated with workspaces

[kylos101]

Is your feature request related to a problem? Please describe

We should test how a node behaves when it is full of workspaces using PVCs, where there is disk activity in the workspaces.

Describe the behaviour you'd like

1. Start a regular workspace in the cluster using PVC on a single node.
2. Cordon the other two nodes
3. Run loadgen, once to fill half of the node, ~9 workspaces
4. Begin to stop the first loadgen run & start second loadgen run with ~20 workspaces (to fill the initial node and trigger scale-up of a new one)

Questions

• What IOPS and bandwidth do we achieve on the initial node? How is it different from a normal cluster?
• How does the regular workspace that was initially started respond as we go through the various phases described above?

Additional context

We're not sure if the IO limiter is needed to cover these disks or not, and whether there will excessive CPU usage as a result of using PVCs for /workspace.

[sagor999]

It worked without issues.
I did not see any IOPS spikes in grafana as well.
Tested with loadgen with 20 workspaces, then with 30, then with 100. (20 and 30 with stop and start at the same time).

[kylos101]

Thanks, @sagor999 ! Could you share What IOPS and bandwidth were achieved? I'm curious to compare with what we have now.

I did not see any IOPS spikes in grafana as well.

Sweet!! Could you share a link to the Grafana and some pictures for posterity? Just super excited, and am glad the test worked out well. 😁

[kylos101]

Also, @sagor999 could you add this to our project and update the status? 🙏

[sagor999]

IOPS:

Large spikes are from RAID device when node start ups I believe (pulling images my guess).
PVC disks seem to have around 100-200 iops on them (some workspaces were stress-ng).

[kylos101]

Thanks, @sagor999 . It looks like when a node is saturated with PVC workspaces and has high R/W data, that system load also goes up (which is expected because we're not limiting I/O on each PVC).

How is CPU usage when there's heavy disk R/W? Can you share a link to the Node Resource Usage for the above node from Grafana and attach a picture to this issue for posterity? It should still be present because it is from 11 days ago.

Aside from the ☝ concern, I cannot think of anything else warranting keeping this issue open. Once you've got that in here, assuming you see no issue with CPU, please close the issue. However, if you do see an issue, please loop in @Furisto to help examine.

[kylos101]

@sagor999 is there anything left to do for stress testing individual nodes?

[sagor999]

I don't think so, plus individual PVCs should be less taxing on the system then massive RAID drives that we have currently. 🙏 (especially that we use software RAID)

[kylos101]

Okay, thank you @sagor999 ! Closing. 😁