Cannot restart workspace (cannot restore backup: permission denied tar)

[WVerlaek]

Bug description

Workspace fails to restart after being stopped.
Full error is:

cannot restore backup:
    github.com/gitpod-io/gitpod/content-service/pkg/initializer.InitializeWorkspace
        github.com/gitpod-io/gitpod/content-service@v0.0.0-00010101000000-000000000000/pkg/initializer/initializer.go:473
  - tar /dst: tar /dst: exit status 2;tar: xyz: Cannot mkdir: Permission denied
    tar: xyz: Cannot mkdir: Permission denied
    ...
    tar: Exiting with failure status due to previous errors:
    github.com/gitpod-io/gitpod/ws-daemon/pkg/content.(*remoteContentStorage).Download
        github.com/gitpod-io/gitpod/ws-daemon/pkg/content/initializer.go:426

Link to log: https://cloudlogging.app.goo.gl/hi4BctLqwXXiuiVt7

Happened twice in the past 30 days.

Previously investigated and closed in: #11183

Steps to reproduce

Unknown

Workspace affected

5125c48a-50d1-4596-92c0-92b1044814a4

Expected behavior

Workspace restarts and restores from backup successfully

Example repository

No response

Anything else?

Slack link: https://gitpod.slack.com/archives/C0201TXMV54/p1677836073098199

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[srgwsrgwetgethg]

Having this issue today with a fairly new workspace. It's the first time I'm having this. Full message:

Content initialization failed: cannot restore backup: tar /dst: tar /dst: exit status 2;tar: .gitpod/.shell_history: Cannot open: Permission denied tar: .gitpod/cmd-0: Cannot open: Permission denied tar: .gitpod/cmd-1: Cannot open: Permission denied tar: .gitpod/prebuild-log-0: Cannot open: Permission denied tar: .gitpod/prebuild-log-1: Cannot open: Permission denied tar: .gitpod/supervisor-termination.log: Cannot open: Permission denied tar: Exiting with failure status due to previous errors

[srgwsrgwetgethg]

Cannot make a new workspace with the same prebuild. The prebuild seems to be problem. Same error message.
Rerunning the prebuild and then creating a new workspace does not help. This is now a serious issue. Can't work now.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.