openfdat complain `permission denied` when restore a content from prebuild

[utam0k]

Bug description

The following error sometimes occurs

cannot initialize workspace: cannot initialize workspace: prebuild initializer: prebuild initializer: openfdat /dst/.docker-root/overlay2/00cec6caf3ff6b85d1d2b27a1413d34466138b3bb3e48202fcd1fdd63dc384f6-init/work/work: permission denied

logs: https://cloudlogging.app.goo.gl/MdRxxSmkQ17f8f6f7
Potentially related issue: #10108

Steps to reproduce

I'm not sure

Workspace affected

n/a

Expected behavior

Prebuild and the content restore works fine

Example repository

No response

Anything else?

No response

Front conversations

[utam0k]

I'm not sure but openfdat may mean golang's function
https://github.com/golang/go/blob/2cfbef438049fd4c3f73d1562773ad1f93900897/src/os/removeall_at.go#L97

[kylos101]

@utam0k I've scheduled this as part of groundwork, nice find.

I found this issue today too, for a ws-daemon where the backing node left our cluster, but, some workspace pods were left behind.

Here are related logs, and a suspect log entry. As you can see here, ws-manager and ws-daemon tried to restart this particular overlay file system many times, but failed.

The workspace I found.

[raphaeltm]

I just want to flag that I'm having this problem as well. It only affects one of our projects, but happens both when we run init tasks, or if we don't run init tasks but run the same commands in a running workspace, then let it shut down and try to re-open it (though I think the error is slightly different in that case). Makes it essentially impossible to work on that project.

[kylos101]

I just want to flag that I'm having this problem as well. It only affects one of our projects, but happens both when we run init tasks, or if we don't run init tasks but run the same commands in a running workspace, then let it shut down and try to re-open it (though I think the error is slightly different in that case). Makes it essentially impossible to work on that project.

hi @raphaeltm 👋 , can you share sample repo where we're able to consistently recreate this problem? 🙏

[raphaeltm]

Hey @kylos101, I've only seen it with one of our projects and I don't have the time to figure out what in the project is causing the problem to create a sample repo. I'd be down to hop on a call and walk you through it, though, if that's helpful?