[self-signed] workspace creation fails in `release-2022.09.0`

[Pothulapati]

Bug description

On a self-signed instance of GItpod (for example local-preview), The workspace creation fails with a CreateConfigError in Kubernetes. Digging more on that we see the following error in Kubernetes

/ # kubectl describe pod  | grep 2148
      NODE_EXTRA_CA_CERTS:              <set to the key 'f0f9c6ddcfb588904de6982b658e17a8cf7b9e3c108fd1380aa2f5d5dfed2148' in secret 'ws-9deee364-eedc-4361-b6b0-90daaa29e0a5'>  Optional: false
  Warning  Failed     7m46s (x12 over 10m)  kubelet            Error: couldn't find key f0f9c6ddcfb588904de6982b658e17a8cf7b9e3c108fd1380aa2f5d5dfed2148 in Secret default/ws-9deee364-eedc-4361-b6b0-90daaa29e0a5

This seems to because of NODE_EXTRA_CA_CERTS being marked as a protected secret but still not rightly loaded into the workspace secret as a key. The same can be confirmed by comparing the secret vs the workspace

Workspace ENV Var

      - name: VSX_REGISTRY_URL
        valueFrom:
          secretKeyRef:
            key: 6271376496bf05a98eb6f9231ef39c87d25a936de821916483e5755bf4dfbe1a
            name: ws-9deee364-eedc-4361-b6b0-90daaa29e0a5
      - name: THEIA_SUPERVISOR_TOKENS
        valueFrom:
          secretKeyRef:
            key: 3a3b612491764cccf7a5bccafc73031bb965cad1592bf600f2c3edca7c6e41bb
            name: ws-9deee364-eedc-4361-b6b0-90daaa29e0a5
      - name: GITPOD_INTERVAL
        value: "30000"
      - name: GITPOD_MEMORY
        value: "524"
      - name: NODE_EXTRA_CA_CERTS
        valueFrom:
          secretKeyRef:
            key: f0f9c6ddcfb588904de6982b658e17a8cf7b9e3c108fd1380aa2f5d5dfed2148
            name: ws-9deee364-eedc-4361-b6b0-90daaa29e0a5
      image: reg.preview.gitpod-self-hosted.com:20000/remote/9deee364-eedc-4361-b6b0-90daaa29e0a5

Secret

Name:         ws-9deee364-eedc-4361-b6b0-90daaa29e0a5
Namespace:    default
Labels:       app=gitpod
              component=workspace
              gitpod.io/workspaceClass=default
              gpwsman=true
              headless=false
              metaID=gitpodio-gitpodmicrosof-uzxl3qjx9yk
              owner=98885113-240a-4499-9923-85cad4ab64a9
              project=
              team=
              workspaceID=9deee364-eedc-4361-b6b0-90daaa29e0a5
              workspaceType=regular
Annotations:  <none>

Type:  Opaque

Data
====
6271376496bf05a98eb6f9231ef39c87d25a936de821916483e5755bf4dfbe1a:  47 bytes
initializer.composite.0.git:                                       40 bytes
3a3b612491764cccf7a5bccafc73031bb965cad1592bf600f2c3edca7c6e41bb:  1596 bytes

Steps to reproduce

Run

docker run -p 443:443 --privileged --name gitpod -it  --mount type=volume,source=gitpod,destination=/var/gitpod eu.gcr.io/gitpod-core-dev/build/local-preview:release-2022.09.0-lp.3

and see that workspace creation fails (while the image builds work)

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

• Slack Thread
• Another Slack Thread

[kylos101]

Hi @Pothulapati thank you for the report!

Customers can workaround this by disabling the feature at install time with a patch like so:

experimental:
  workspace:
    enableProtectedSecrets: false

However that does carry some risk (outlined here).

As background, we enabled protected secrets by default here.

[kylos101]

@Pothulapati it looks like this is blocking local-preview. Have you been able to confirm if it is blocking a single cluster ref arch that is using self-signed certs?

[Pothulapati]

👋 Good Morning, @kylos101 !

Currently, I can only confirm that it fails in local-preview which is a self-signed instance, but I am verifying the same in self-signed AWS now. Will update once I get this up! 👀

from what I can tell, this is blocking local-preview, but so far, no customers have bumped into this issue.

Correct. Because we haven't done a local-preview release yet so no user effect yet. We should add the configuration that you specified if we plan to do a release.

[Pothulapati]

@kylos101 Can confirm that the issue also happens with self-signed EKS instance. I have one at https://3c759-aws.tests.gitpod-self-hosted.com/workspaces, but you can get the same by running

werft run github -a skipTests=true -a selfSigned=true -a preview=true -j .werft/eks-installer-tests.yaml

[kylos101]

Okay, thanks @Pothulapati . 😬 For the meantime, I think we'll need to recommend the ☝️ workaround till a fix is pushed. Would it be appropriate to add a comment in the September Upgrade instructions?

I've added to our groundwork, but, it may not land till the next self-hosted release. Let us know your thoughts? The trade-off of using the workaround (disabling protected_secrets) is the related security risks.

[kylos101]

Hi there, @utam0k is going to take a look at this issue, to see if we can patch the September release.

cc: @lucasvaltl @corneliusludmann @Pothulapati