[PVC] orphan PVC left if the ws-manager unable to start workspace pod

[jenting]

Bug description

Two scenarios:

• PVC bound, but workspace pod gone: the PVC object exists even though the workspace pod is gone.
  The orphan PVC object still exists within the cluster even though the workspace pod is gone. And the ws-manager with errors

  "error":"timed out waiting for the condition","instanceId":"ea1e5ba2-54d6-4611-b8a6-ba8e9d791cb7","level":"warning","message":"was unable to start workspace","pod":"ws-ea1e5ba2-54d6-4611-b8a6-ba8e9d791cb7"

• PVC bound and workspace pod Terminating: the PVC state from pending -> bound after one hour because of the GCP limitation, but the workspace pod timed out. The workspace pod is Terminating, and PVC bound.

Steps to reproduce

• PVC bound, but workspace pod Pending: the PVC object exists even though the workspace pod is gone
  When running the loadgen with 200 workspaces simultaneously (100 regular workspaces + 100 regular workspaces + PVC). After testing is done, there is some orphan PVC left, and the ws-manager does not handle it. Reference code
  

  gitpod/components/ws-manager/pkg/manager/manager.go

  Line 338 in ba74cdb

  clog.WithError(err).WithField("req", req).WithField("pod", pod.Name).Warn("was unable to start workspace")

• PVC pending and workspace pod Pending: the PVC pending -> bound after one hour, but the workspace pod timed out. The workspace pod is pending, and PVC bound
  [PVC] orphan PVC left if the ws-manager unable to start workspace pod #13282 (comment)

Workspace affected

No response

Expected behavior

The workspace pod and PVC object should be removed.

Example repository

No response

Anything else?

#7901

[kylos101]

@jenting can you share why this moved back from In-Validation, and what the plan is for next steps here, in this issue? Also, this seems like it will be a blocker to releasing PVC to a larger group (other than Gitpodders). cc: @sagor999

[jenting]

The original solution has side effect that the workspace can not be terminated because we delete PVC object too early which makes the PVC in terminated state (thank to finalizer). However, when snapshot controller takes snapshot, it adds finalizer to PVC object but it fails because the finalizer can't add when object is terminated. So, the snapshot failed.

I think it's not blocker because the PVC never be mounted by the workspace pod, and the PVC object is in Pending state. What we left is we need to garbage collection the Pending PVC object.

[kylos101]

I see, thank you, @jenting .

[jenting]

Another scenario is the PVC in a Pending state within one hour because of the GCP limitation, and then the PVC bound because the limitation is gone.

However, the workspace pod timed out, causing it to be in a Terminating state but PVC in a bound state.
The ws-manager should garbage collect the workspace pod and PVC. For this case, we don't need to take the volume snapshot. Related private gist.