Adjust URL Being Used To Fetch .gitpod.yml Files From GitHub Enterprise Instances

[jimmybrancaccio]

Bug description

I currently have a self-hosted instance deployed which appears to be ignore the image: image-name line in my .gitpod.yml file within a repository. For example in my .gitpod.yml I've tried:

image: node:16.14-buster

or

image: jimmyb/workspace-laravel

but every time the gitpod/workspace-full image is loaded. I captured some logs from the build process and as you can see it's pulling the workspace-full image - https://ghostbin.linuxbox.ninja/paste/6jyt8

I've tested this on gitpod.io and I cannot replicate the issue there. @mrsimonemms was also unable to replicate the issue on his self-hosted instance so it sounds like an issue specific to my instance.

Last night I tore down my Kubernetes cluster and rebuilt it from scratch, but the same issue occurs. I am currently on main.2860 which I updated to from main.2847 (where the issue existed for me) to no avail.

Is there some sort of setting I can look at that would force the workspace-full image to be used? I know if there is no .gitpod.yml or image set it will default to the workspace-full image, but that's definitely not the case here, I am setting the image to be used, it's just being ignored.

To further troubleshoot, I set my .gitpod.yml up as:

image: jimmyb/workspace-laravel

vscode:
  extensions:
    - esbenp.prettier-vscode

I started a new workspace, it ignored the image line and used the workspace-full image, but it did install the extension. So it seems it's not completely ignoring the .gitpod.yml file.

My cluster is built from k3s, using 3 Ubuntu 20.04.x VMs (1 controller, 2 workers). If any other information about the cluster is needed, please don't hesitate to ask!

Any help on where to look next would be greatly appreciated!

Steps to reproduce

So here's the thing, I/we can't seem to reproduce this 😩

Workspace affected

No response

Expected behavior

I would expect that the image specified in my .gitpod.yml file be used for my workspace instead of the gitpod/workspace-full image.

Example repository

No response

Anything else?

No response

[jimmybrancaccio]

Discord thread at - https://discord.com/channels/816244985187008514/963419914809528340

[corneliusludmann]

Anyone from team @gitpod-io/engineering-workspace some hints on how to debug this?

[princerachit]

@jimmybrancaccio I can't promise that I know the resolution but I can take a look if you share the following with me:

• Public repo on which you are able to reproduce this issue
• Installer generated yaml files (redacted), specifically I want to see configmaps related to image builder

[jimmybrancaccio]

Hey @princerachit!

• I don't have the repo public, it's sitting on my GitHub Enterprise instance. That said, there's nothing really special about the repo. I just have a single file in it, gitpod.yml with the contents:

image: jimmyb/workspace-laravel

Hopefully this is the correct information for the configmap for the image-builder. If it's not don't hesitate to let me know, as well as if there's anything else I can provide!

apiVersion: v1
data:
  image-builder.json: |-
    {
      "orchestrator": {
        "wsman": {
          "address": "ws-manager:8080",
          "tls": {
            "ca": "/wsman-certs/ca.crt",
            "crt": "/wsman-certs/tls.crt",
            "key": "/wsman-certs/tls.key"
          }
        },
        "pullSecret": "builtin-registry-auth",
        "pullSecretFile": "/config/pull-secret.json",
        "baseImageRepository": "registry.my-gitpod-instance.com/base-images",
        "workspaceImageRepository": "registry.my-gitpod-instance.com/workspace-images",
        "builderImage": "eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:0d7a2c9dd3c88afe9f843243c29595dcc8242757"
      },
      "refCache": {
        "interval": "6h0m0s",
        "refs": [
          "docker.io/gitpod/workspace-full:latest"
        ]
      },
      "service": {
        "address": ":8080",
        "tls": {
          "ca": "",
          "crt": "",
          "key": ""
        }
      },
      "prometheus": {
        "address": "127.0.0.1:9500",
        "tls": {
          "ca": "",
          "crt": "",
          "key": ""
        }
      },
      "pprof": {
        "address": ":6060"
      }
    }
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: gitpod
    meta.helm.sh/release-namespace: gitpod
  creationTimestamp: "2022-04-11T22:13:36Z"
  labels:
    app: gitpod
    app.kubernetes.io/managed-by: Helm
    component: image-builder-mk3
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:image-builder.json: {}
      f:metadata:
        f:annotations:
          .: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
        f:labels:
          .: {}
          f:app: {}
          f:app.kubernetes.io/managed-by: {}
          f:component: {}
    manager: helm
    operation: Update
    time: "2022-04-11T22:13:36Z"
  name: image-builder-mk3-config
  namespace: gitpod
  resourceVersion: "11395"
  uid: 20f4f313-61f7-4dac-a82e-05f46f7a0d2e

[princerachit]

@jimmybrancaccio I asked for a public repo because I suspected that a special character(hidden) might be causing the issue (totally wild guess). Can you create and share a public repo with same .gitpod.yml configuration just so that I don't miss anything.

Also, what do you see in d_b_workspace table for this workspace?

SELECT * from d_b_workspace where id = 'my-workspace-name'

[jimmybrancaccio]

@princerachit Ah, that's very interesting about the special character. I can confirm there are none, in fact I first noticed this issue with two different repositories that I know have worked in Gitpod and used the image I specified in the .gitpod.yml file. I had ended up creating the gitpod-test repository as part of my troubleshooting (so I could have the bare minimum to create a workspace).

I forgot I had created this test repository on GitHub - https://github.com/jimmybrancaccio/gitpod-test

For the entry in MySQL:

*************************** 3. row ***************************
                   id: jimmybrancaccio-gitpodte-iys8rb8wtqq
         creationTime: 2022-04-12T15:26:54.445Z
              ownerId: 083f755f-fe74-4779-85ea-2f2026256776
           contextURL: https://my-github-enterprise.com/jimmybrancaccio/gitpod-test
          description: jimmybrancaccio/gitpod-test - master
              context: {"isFile":false,"path":"","title":"jimmybrancaccio/gitpod-test - master","ref":"master","refType":"branch","revision":"92cd51664b067a4fe5b1ec7b54b79dae6149aca5","repository":{"cloneUrl":"https://my-github-enterprise.com/jimmybrancaccio/gitpod-test.git","host":"my-github-enterprise.com","name":"gitpod-test","owner":"jimmybrancaccio","private":false},"normalizedContextURL":"https://my-github-enterprise.com/jimmybrancaccio/gitpod-test","checkoutLocation":"gitpod-test"}
               config: {"ports":[{"port":3000}],"tasks":[],"image":"docker.io/gitpod/workspace-full:latest","_origin":"default"}
          imageSource: {"baseImageResolved":"docker.io/gitpod/workspace-full:latest"}
    imageNameResolved: registry.my-gitpod-instance.com/workspace-images:37fa6b72cbb0b2b1459a59acbe862f31a820b2db62d740d3ac57fb93d357526a
             archived: 0
            shareable: 0
        _lastModified: 2022-04-12 15:26:54.895996
              deleted: 0
                 type: regular
baseImageNameResolved: docker.io/gitpod/workspace-full@sha256:b484ad49d47bf70776de1d1333379d6b5f1d56c0a3fc1f31a24eb273ef772e3e
          softDeleted: NULL
               pinned: 0
      softDeletedTime: 
   contentDeletedTime: 
    basedOnPrebuildId: 
    basedOnSnapshotId: 
            projectId:

[jimmybrancaccio]

I took a look at server component logs today and I did notice this, I wonder if it could be related?

{
    "component": "server",
    "severity": "DEBUG",
    "time": "2022-04-14T16:14:27.658Z",
    "context": { "userId": "083f755f-fe74-4779-85ea-2f2026256776" },
    "message": "Config string undefined, using default config",
    "payload": {
        "repoCloneUrl": "https://github-enterprise-url.com/jimmybrancaccio/gitpod-test.git",
        "revision": "92cd51664b067a4fe5b1ec7b54b79dae6149aca5"
    }
}

[jimmybrancaccio]

Ok, so I'm thinking this is related to me using GitHub Enterprise. I created an app on GitHub.com and added that as an integration on my Gitpod instance. I then spun up a workspace using https://github.com/jimmybrancaccio/gitpod-test and what do you know, it worked fine and used the image I had defined in .gitpod.yml.

In my image I am using NodeJS v16.14.2 as well as I installed the Laravel and PHP_Codesniffer Composer packages which I see (so I know it's using my defined image):

gitpod /workspace/gitpod-test (main) $ ls -la ~/.config/composer/vendor/bin/
total 20
drwxr-xr-x 2 gitpod gitpod 4096 Apr 10 21:38 .
drwxr-xr-x 8 gitpod gitpod 4096 Apr 10 21:38 ..
-rwxr-xr-x 1 gitpod gitpod 3221 Apr 10 21:38 laravel
-rwxr-xr-x 1 gitpod gitpod 3242 Apr 10 21:38 phpcbf
-rwxr-xr-x 1 gitpod gitpod 3239 Apr 10 21:38 phpcs

So now we need to figure out why... 🤔

[jimmybrancaccio]

Could it be related to #9126?

[princerachit]

@gitpod-io/engineering-webapp based on the findings above by @jimmybrancaccio do you think this is something that is under your expertise?

[geropl]

@jankeromnes Could you have a look into this? Relevant passage from above:

Ok, so I'm thinking this is related to me using GitHub Enterprise. I created an app on GitHub.com and added that as an integration on my Gitpod instance.