self-hosted environment workspace took a long time to load completion

[LyleLaii]

Bug description

As far as I can see, my workspace takes more than 5 minutes to load the file icon theme and the gitpod feature list.

Kots installed with self-signed ca in an air gappped network and version is release-2022.05.2.5, use workspace-full image.

after workspace load completion, I cannot see gitpod feature list, or before refreshing the webpage, it is disabled.

And there are some errors in the browser console.

after this warn message print, refresh the webpage, it load completion.

and also print some certificate errors.

Looks like something to do with github authentication? But I integrated self-hosted gitlab instead of github authentication.

Please let me know if need more information, thanks!

Steps to reproduce

1. an air gappped network
2. Install kots
3. modify registry setting to use docker registry(harbor)
4. install gitpod with self-signed certificate
5. create a workspace

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[lucasvaltl]

Hey @LyleLaii - thanks for raising this and putting in the time to write such a detailed and helpful description of your issue. It seems that you are having issues with the VS Code extension (which populates the feature list) which is taking a long time load.

I will forward this to the team (team IDE) that is responsible and they can take a look at this.

One thing that would help us debug here: Can you check wether you have an error in the logs for the ws-proxy component have an error in the form of could not connect to backend /static/out/vs/loader.js: x509: certificate signed by unknown authority? If this is the case we might have fixed this via the PR mentioned below.

@team IDE:

• We recently fixed something that might be related here - [installer] add customCACert volume to ws-proxy #10850. This should be included in next weeks self-hosted release. This might be related - see above.

[LyleLaii]

@lucasvaltl Sorry for the long time no reply. I've added my cacert inside the ws-proxy service using a rebuild image when I deployed, and it looks like this is all the logs since the deployment.

{"level":"warning","message":"config value 'workspacePodConfig.supervisorImage' is deprected, use it only to be backwards compatible","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"WARNING","time":"2022-06-17T03:11:03Z"}
{"addr":"127.0.0.1:9500","level":"info","logger":"controller-runtime.metrics","message":"Metrics server is starting to listen","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","time":"2022-06-17T03:11:04Z"}
{"level":"info","message":"workspace info provider started","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","time":"2022-06-17T03:11:04Z"}
{"addr":":60060","level":"info","message":"serving pprof service","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","time":"2022-06-17T03:11:04Z"}
{"level":"info","message":"started proxying on :8080","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","time":"2022-06-17T03:11:04Z"}
{"level":"info","message":"🚪 ws-proxy is up and running","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","time":"2022-06-17T03:11:04Z"}
{"addr":"{\"IP\":\"127.0.0.1\",\"Port\":9500,\"Zone\":\"\"}","kind":"metrics","level":"info","message":"Starting server","path":"/metrics","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","time":"2022-06-17T03:11:04Z"}
{"addr":"{\"IP\":\"::\",\"Port\":8086,\"Zone\":\"\"}","kind":"health probe","level":"info","message":"Starting server","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","time":"2022-06-17T03:11:04Z"}
{"level":"info","logger":"controller.pod","message":"Starting EventSource","reconciler group":"","reconciler kind":"Pod","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","source":"kind source: *v1.Pod","time":"2022-06-17T03:11:04Z"}
{"level":"info","logger":"controller.pod","message":"Starting Controller","reconciler group":"","reconciler kind":"Pod","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","time":"2022-06-17T03:11:04Z"}
{"level":"info","logger":"controller.pod","message":"Starting workers","reconciler group":"","reconciler kind":"Pod","serviceContext":{"service":"ws-proxy","version":"commit-922c9e8fceace4abc3ed37e92baaee6f10e6a4b7"},"severity":"INFO","time":"2022-06-17T03:11:04Z","worker count":1}
W0618 02:08:17.634721       1 reflector.go:442] k8s.io/client-go@v0.23.5/tools/cache/reflector.go:167: watch of *v1.Pod ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
I0618 02:08:31.843880       1 trace.go:205] Trace[1276116358]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.23.5/tools/cache/reflector.go:167 (18-Jun-2022 02:08:18.861) (total time: 12982ms):
Trace[1276116358]: ---"Objects listed" error:<nil> 12981ms (02:08:31.843)
Trace[1276116358]: [12.982307585s] [12.982307585s] END

[akosyakov]

@LyleLaii Could you follow this https://www.gitpod.io/docs/troubleshooting#gitpod-logs-in-vs-code-web-and-desktop to share more detail with us?

You mentioned that you are in air-gapped network. Gitpod needs access to https://open-vsx.org/ and https://openvsxorg.blob.core.windows.net/ to install extension from your machine as well as from OpenVSX proxy running within Gitpod. Maybe some network configuration is missing on your side?

[akosyakov]

About github token: Do you open GitHub repo with GitLab auth somehow? 🤔