SSH connection fails if workspace image does not contain "/home/gitpod" directory and "gitpod" user

[Acepie]

Bug description

We very recently ran into a weird issue when trying the new direct ssh connection with public/private keys. The behavior is that when trying to connect to the workspace (either directly via ssh command or via the vscode desktop integration) it "appears" to succeed authentication but immediately gets a closed by remote host error. After looking in the logs of the workspace pod itself, it turned out that the authentication was attempting to get the /home/gitpod/authorized_keys file despite the fact that the base image for this workspace did not actually include that file/user yet (we needed a tool that is only available internal and were using its build image). I think previously when starting a workspace, if that user/dir did not exist than it was being autocreated but it seems like that autocreation process happens after the ssh auth check?

Steps to reproduce

1. Create a workspace using an image that is not natively setup for gitpod (some basic ubuntu image for example)
2. Add valid ssh public keys to user config if not already done
3. Attempt to use the ssh connection command to ssh
4. Get connection closed by remote host error

Workspace affected

No response

Expected behavior

SSH connection should succeed assuming the public/private key setup is valid

Example repository

No response

Anything else?

No response

[axonasif]

Hi @Acepie, thanks for reporting this issue 👍
I'm not sure if you've tried, as a temporary workaround you could try to symlink your custom home dir to /home/gitpod until this issue is fixed.

[Acepie]

I ultimately worked around the issue by making a new multi stage dockerfile starting from a full workspace and just copied the relevant binaries from our old image into the final image. That resolved my issues

[akosyakov]

@iQQBot is it because we don't check for content readiness while trying to establish ssh connection from ssh gateway to workspace?

I don't think one cannot have /home/gitpod and gitpod user. The supervisor has these assumptions built in all over the place?