[PVC][integration test] restart control plane components when/after volume snapshot in-progess/done

[jenting]

Is your feature request related to a problem? Please describe

Add/modify the integration test to test the control plane component (ws-manager, snapshot-controller) restart when volume snapshot in-progress or after volume snapshot is done.

Describe the behaviour you'd like

• To catch this issue [PVC] ws-manager event workers hang forever once over 100 VolumeSnapshots and ws-manager restart #13007 in the automation test
• To test the regular workspace test case 6 (During the workspace stopping (during PVC backup), restart the control plane components) from manual testing to automation testing

Describe alternatives you've considered

Additional context

#7901

[sagor999]

I tested restarting both snapshot-controller and ws-manager after and during workspace shutdown and both worked flawlessly as far as I can tell.
Also tested ws-manager restart with 150 volume snapshots.

[jenting]

I tested restarting both snapshot-controller and ws-manager after and during workspace shutdown and both worked flawlessly as far as I can tell.

Also tested ws-manager restart with 150 volume snapshots.

Nice 🤩 Thank you @sagor999

[kylos101]

@sagor999 thank you so much!

@jenting the issue description and title confuse me, because they mentions both loadgen and integration tests. I think they will be similar effort...therefore my preference would be to do with integration tests (so we test more often, instead of once a week). Also, this will keep loadgen flexible...if we need to restart ws-manager or snapshot-controller, we can do manually.

[jenting]

I don't have a strong reason to add a restart control plane component on both loadgen and integration tests.

But I was like to catch the incident we met that over 100+ VolumeSnapshot, restart control plane, and create new workspace will hang.
I think it can only be caught if we run loadgen test because the integration test generally would not have stress test on over 100+ VolumeSnapshots. But we could add the manual step that for each loadgen test finished, make sure to restart the control plane components to see if the new workspace can open or not. After that, we could change this issue to work on integration tests only.