Permissions in docker images are lost during gitpod restore

[shaal]

UPDATE:

This problem is actually not just about setuid, but about permissions in general.
Follow this comment #3174 (comment) for an even simpler steps to reproduce the issue.

---

Describe the bug

In a workspace using sudo docker-up and pulling a docker image, it has sudo with the correct setuid bit.
-rwsrwxrwx 1 root root 157192 Feb 2 2020 /usr/bin/sudo

But after timeout/restarting the workspace, the sudo in the image has the wrong setuid bit
-rwxr-xr-x 1 root root 157192 Feb 2 2020 /usr/bin/sudo

Steps to reproduce

I created a minimal setup to reproduce the behavior:
https://gitpod.io/#https://github.com/shaal/gitpod-restart-test

1. Start the workspace
2. The workspace will run sudo docker-up
3. The workspace will pull an image and display the image's /usr/bin/sudo permissions (ie. docker run -it --rm drud/ddev-webserver:v1.16.3 ls -l /usr/bin/sudo)
4. The permission should be -rwsrwxrwx
5. Using https://gitpod.io/workspaces - Stop the current workspace (alternatively you can wait for the workspace to timeout)
6. Start the same workspace again
7. Run docker run -it --rm drud/ddev-webserver:v1.16.3 ls -l /usr/bin/sudo
8. Now the permission is wrong -rwxr-xr-x, it's missing the setuid bit.

• Please note that the issue happens with any image.

Expected behavior

The setuid bit should stay the same after restarting a workspace.

Additional information

Example repository

https://github.com/shaal/gitpod-restart-test

[ArthurSens]

Thanks for the report @shaal!

Sorry if the answer is obvious, but how does changing the setuid bit affects your work when using Gitpod? I'm not saying it isn't a bug, just thinking about prioritization... 🤔

[shaal]

Sorry, I tried being specific and wanted to show the simplest way to reproduce the bug.

The way it affects my work, is that when a workspace times out, I can no longer work on that workspace again, until I destroy all docker containers.

You can see an example project here -
https://gitpod.io/#https://github.com/shaal/ddevenv

It opens one terminal with sudo docker-up, and another terminal with ddev start (pulling docker images to build the environment).
When starting the workspace everything works well.
Run gp url 8080, and open that URL in browser.
You would see the following:

Now stop the workspace (or wait for it to timeout)
Start the workspace again.

You'll see in the 3rd terminal, the command ddev start, failing with this error:

Starting ddevenv... 
Building ddev-ssh-agent 
Recreating ddev-ssh-agent ... done
 
Failed to start ddevenv: ddev-ssh-agent failed to become ready; debug with 'docker logs ddev-ssh-agent'; error: container /ddev-ssh-agent unhealthy:  

Running docker logs ddev-ssh-agent display this:

sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set
Creating a proxy socket...
Running socat UNIX-LISTEN:/tmp/.ssh-agent/proxy-socket,perm=0666,fork UNIX-CONNECT:/tmp/.ssh-agent/socket
Launching ssh-agent...
2021/02/11 18:53:12 socat[11] E bind(5, {AF=1 "/tmp/.ssh-agent/proxy-socket"}, 30): Permission denied
unix_listener: cannot bind to path /tmp/.ssh-agent/socket: Permission denied

[rfay]

The fundamental problem is that several of the images used for ddev have sudo in them, and of course sudo is not usable with the setuid bit reset inappropriately. I'm sure there are other utilities that would be hobbled with the setuid inappropriately reset, but sudo is the big one.

When a workspace is awakened, the setuid bit is no longer set on sudo inside these images. It should not have been changed.

And it doesn't matter if you re-pull the image. Same result.

[csweichel]

Thank you for reporting this.

This look like an issue with the content init during workspace startup. Apparently we're not preserving file modes correctly in some cases.

[rfay]

I'm obviously a novice to the codebase, but a common way to lose setuid bits and ownership information is to run tar x without root privileges. I wonder if that's happening here:

gitpod/components/content-service/pkg/archive/tar.go

Line 102 in 4d3947d

tarcmd := exec.Command("tar", "x")

[rfay]

This problem is actually not just about setuid, but about permissions in general.

In a docker image that had a directory with permissions 0777 (/etc/alternatives), after restore it has 0755 privileges.

@shaal maybe you can update the title to add "different permissions entirely"