Unable to start Workspace - stuck at acquiring node with k3s v1.20.0+k3s2

[barnybadzoo]

System Setup:

• Ubuntu 20.04
• k3s v1.20.0+k3s2 with disable=local-storage,servicelb,traefik
• metalllb, ingress-nginx, gitlab, longhorn 1.10

Describe the bug

• Gitpod hang at Acuiring Nodes 1/3
• The Workspace Build was successfully but the new started workspace pod readiness fails
• Within the workspace pod the log stops after vscode.docker@1.47.3 and flood log with 2 UNKNOWN: cannot update status: completeWorkspaceObjects: selfLink was empty:

{"component":"workspace","severity":"info","time":"2021-01-10T23:18:20.050Z","message":"Deploying VS Code extension \"vscode.docker@1.47.3\" from \"/tmp/vscode-extensions/vscode.docker@1.47.3/extension\"","payload":{"name":"root","params":[]}} {"component":"workspace","severity":"info","time":"2021-01-10T23:18:20.052Z","message":"Deploying VS Code extension \"vscode.vscode-theme-seti@1.44.2\" from \"/tmp/vscode-extensions/vscode.vscode-theme-seti@1.44.2/extension\"","payload":{"name":"root","params":[ ]}} {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"jsonrpc2: code -32603 message: Request openPort failed with message: 2 UNKNOWN: cannot update status: completeWorkspaceObjects: selfLink was empty, can't make reference","level":"warning","message":"cannot auto-expose port","port":{"Served":false,"Exposed":false,"Visibility":1,"URL":"","OnExposed":3,"LocalhostPort":3000,"GlobalPort":3000},"serviceContext":{"service":"supervisor","version":""},"severity":"WARNING","tim e":"2021-01-10T23:18:20Z"} {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"jsonrpc2: code -32603 message: Request openPort failed with message: 2 UNKNOWN: cannot update status: completeWorkspaceObjects: selfLink was empty, can't make reference","level":"warning","message":"cannot auto-expose port","port":{"Served":false,"Exposed":false,"Visibility":1,"URL":"","OnExposed":3,"LocalhostPort":3000,"GlobalPort":3000},"serviceContext":{"service":"supervisor","version":""},"severity":"WARNING","tim e":"2021-01-10T23:18:22Z"}

Steps to reproduce

• install self-hosted beta5 at k3s v1.20.0+k3s2 with helm chart
• use kube.domain.com as hostname
• minio and registry enabled in chart
• wsdaemon setting:
• socket: /run/k3s/containerd/containerd.sock
• noderRoots: /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io

All Pods should now be installed without errors.

• Generate a wildcard ingress for gitpod:

`kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
name: gitpod-ingress
namespace: gitpod
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.provider: nginx
nginx.ingress.kubernetes.io/proxy-body-size: 512m
nginx.ingress.kubernetes.io/proxy-connect-timeout: '3600'
nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: ""
spec:
tls:
- hosts:
- "gitpod.kube.domain.com"
- "registry.gitpod.kube.domain.com"
- "minio.gitpod.kube.domain.com"
- ".ws.gitpod.kube.domain.com"
secretName: wildcard-cert

rules:
- host: gitpod.kube.domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: proxy
port:
number: 443
- host: registry.gitpod.kube.domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: registry
port:
number: 443
- host: minio.gitpod.kube.domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: proxy
port:
number: 443
- host: "*.ws.gitpod.kube.domain.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: proxy
port:
number: 443 `

Start a Gitpod Session through gitlab, which results in the acuiring node error i described above

Expected behavior

The workspace pod should start normal

[csweichel]

The workspace seems to hang in Acquiring Node because the status update mechanism of Gitpod breaks here. It would seem that trying to get the pod from Kubernetes fails because client-go finds that selfLink was empty, can't make reference.

I'd assume that something like kubectl get pods works just fine in your setup?

Without having checked this, considering that everything else in your cluster seems to work just fine (please correct me if I'm wrong), it might have to do with outdated client-go libraries (1.15) within Gitpod - we're looking to fix that (#2689).

[barnybadzoo]

everything else in the cluster works fine.. also kubectl get pods..

I also tried to install self-hosted beta5 on belower k3s version... I couldn't get one of them to work.
i tried 1.19.5+k3s2 and v1.18.13+k3s1.. but this might be also in issue with longhorn .

Can you clarify the error messages for me ?
Did the setup of vscode.docker@1.47.3 broke due outdated go-libary ?

I am testing here all day.. So let me know if i can help..
When will a Patch be available ?

Also one thing i would like to know is, why you distribute the workspace pod as "pod", and not as "deployment/statefulset".
The pod cant be edited or updated easily with new settings (for testing settings like hostNetwork:true or different startup script.. For testing i did export it and recreated it everytime.. A deployment/statefulset could be easily changed and redeployed..

[corneliusludmann]

The original issue seems to be a duplicate of #2689 which is closed already. Can this issue be closed as well?

[barnybadzoo]

Current Version (tried master.405) installs without problem, but i receive error 404 by accessing gitpod. I am still searching for a solution...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.