Permission denied when accessing .docker-root/overlay2

[sagor999]

Bug description

I see a lot (40k in last 3 hours) errors like this in SaaS:

jsonPayload: {
@type: "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent"
error: "permission denied"
gid: 33333
level: "warning"
message: "cannot chown"
path: ".docker-root/overlay2/205df79f676c4bbc3e8af627dc2a78228e55f494d7b21c5725fcaeb808e285cb/diff/home/node/parse-server/"
uid: 33333

Link to logs: https://cloudlogging.app.goo.gl/mZg3N1J2w58wm7fLA

This might be related to an issue customer is having as well:
https://app.frontapp.com/inboxes/teammates/9932874/inbox/open/41825492426

Potentially related issue: #9263

Would like to understand why we are getting those errors and what is the impact.

Steps to reproduce

One potential repro case:
This seems to do with running docker-compose pull and docker-compose build in init task.

Workspace affected

tracthq-tractweb-xtjbxw26gqh

Expected behavior

We shouldn't be getting permission denied errors when accessing .docker-root I think.

Example repository

No response

Anything else?

No response

[sagor999]

@csweichel would you have any insight as to why we might be getting this error?

[utam0k]

I guess this problem is related to seccomp notify.

gitpod/components/workspacekit/pkg/seccomp/notify.go

Lines 426 to 451 in e24461c

	func (h *InWorkspaceHandler) Chown(req *libseccomp.ScmpNotifReq) (val uint64, errno int32, flags uint32) {
	log := log.WithFields(map[string]interface{}{
	"syscall": "bind",
	"pid": req.Pid,
	"id": req.ID,
	})
	memFile, err := readarg.OpenMem(req.Pid)
	if err != nil {
	log.WithError(err).Error("cannot open mem")
	return
	}
	defer memFile.Close()
	pth, err := readarg.ReadString(memFile, int64(req.Data.Args[0]))
	if err != nil {
	log.WithError(err).Error("cannot open mem")
	return
	}
	if strings.HasPrefix(pth, "/dev/pts") {
	return 0, 0, 0
	}
	return 0, 0, libseccomp.NotifRespFlagContinue
	}

[kylos101]

@utam0k after looking at #9247, maybe consider this issue?

[sagor999]

From another customer via Front:

gitpod.yml is as following:
------------------
tasks:
- init: |
sed -i 's/git@github.com:celtra/https:\/\/github.com\/celtra/g' .gitmodules
git submodule init
git submodule sync
git submodule update --recursive

git clone https://github.com/celtra/uab-devcontainers.git .devcontainer

touch .env .test-env
time docker-compose build
------------------

Prebuild environment successfully builds, but when trying to create a new workspace with it, the following error is shown:

------------------
Oh, no! Something went wrong!
cannot initialize workspace: cannot initialize workspace: prebuild initializer: prebuild initializer: openfdat /dst/.docker-root/overlay2/1htqhv8btix39teqczhlqgcbp/work/work: permission denied
------------------

[utam0k]

This matter has appeared since May 5 when the gen44 was shipped.
https://cloudlogging.app.goo.gl/xCD6MPnXVcY55Fxj6

[utam0k]

Possibly related:

• [docker-up] Update docker compose to 2.5.0 #9699
• [content-service] Fix backup restore #9732

[utam0k]

It appears that I do not have enough permissions to set the attribute user.overlay.impure.
https://cloudlogging.app.goo.gl/dQBLSUSoQboBXVPn9

gitpod/components/content-service/pkg/archive/tar.go

Line 192 in 8c50daf

log.WithField("name", key).WithField("value", value).WithField("file", name).WithError(err).Error("restoring extended attributes")