Closed as not planned
Description
Bug description
Workspaces do not initialize because they cannot connect to ws-daemon. As a result, users were unable to start workspaces in the US region.
kyle@work:~$ kubectl get pod -l metaID=gitpodio-templatepython-vlhvnj7u60f
NAME READY STATUS RESTARTS AGE
ws-64bdc59e-d708-4304-83ec-0ed5fa6c9797 0/1 Running 0 51s
kyle@work:~$ kubectl get pod -l metaID=gitpodio-templatepython-vlhvnj7u60f
NAME READY STATUS RESTARTS AGE
ws-64bdc59e-d708-4304-83ec-0ed5fa6c9797 0/1 Error 0 2m29s
kyle@work:~$ kubectl logs ws-64bdc59e-d708-4304-83ec-0ed5fa6c9797
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"socket did not appear before context was canceled: errors of connect to ws-daemon","level":"error","message":"cannot connect to daemon from ring0","ring":0,"serviceContext":{"service":"workspacekit","version":"commit-8d009263e19017f630e6f12436d3aa9186b8355c"},"severity":"ERROR","time":"2022-09-13T09:40:47Z","workspaceId":"gitpodio-templatepython-vlhvnj7u60f"}
{"level":"info","message":"ring0 stopped","ring":0,"serviceContext":{"service":"workspacekit","version":"commit-8d009263e19017f630e6f12436d3aa9186b8355c"},"severity":"INFO","time":"2022-09-13T09:40:47Z","workspaceId":"gitpodio-templatepython-vlhvnj7u60f"}
kyle@work:~$ kubectl describe pod ws-64bdc59e-d708-4304-83ec-0ed5fa6c9797
Name: ws-64bdc59e-d708-4304-83ec-0ed5fa6c9797
Namespace: default
Priority: 0
Node: workspace-ws-us64-internal-xl-pool-9xqw/10.10.0.15
Start Time: Tue, 13 Sep 2022 05:38:42 -0400
Labels: app=gitpod
component=workspace
gitpod.io/networkpolicy=default
gitpod.io/workspaceClass=g1-large
gpwsman=true
headless=false
metaID=gitpodio-templatepython-vlhvnj7u60f
owner=8df3495b-685d-46e0-9820-009cc3b4afd8
project=
team=
workspaceID=64bdc59e-d708-4304-83ec-0ed5fa6c9797
workspaceType=regular
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: false
cni.projectcalico.org/containerID: 80dd8a3d4afb56b161b3b0503ff31f25f2bbadc3f552d5f900255be4991eda71
cni.projectcalico.org/podIP:
cni.projectcalico.org/podIPs:
container.apparmor.security.beta.kubernetes.io/workspace: unconfined
gitpod.io/cpuBurstLimit: 8
gitpod.io/cpuMinLimit: 3
gitpod/admission: admit_owner_only
gitpod/contentInitializer: <removed>
gitpod/customTimeout: 60m
gitpod/id: 64bdc59e-d708-4304-83ec-0ed5fa6c9797
gitpod/imageSpec:
<removed>
gitpod/never-ready: true
gitpod/ownerToken: <removed>
gitpod/servicePrefix: gitpodio-templatepython-vlhvnj7u60f
gitpod/url: https://gitpodio-templatepython-vlhvnj7u60f.ws-us64.gitpod.io
kubernetes.io/egress-bandwidth: 300M
kubernetes.io/ingress-bandwidth: 300M
seccomp.security.alpha.kubernetes.io/pod: localhost/workspace_default_main.4550.json
Status: Failed
IP: 10.20.255.224
IPs:
IP: 10.20.255.224
Containers:
workspace:
Container ID: containerd://b23a73fa033d2829c672631672033ded80d1a06f1f75b256f17254c0a8fbfe5d
Image: reg.ws-us64.gitpod.io:20000/remote/64bdc59e-d708-4304-83ec-0ed5fa6c9797
Image ID: reg.ws-us64.gitpod.io:20000/remote/64bdc59e-d708-4304-83ec-0ed5fa6c9797@sha256:6e5e9b10a478e0f175a1a98e68734a27bffdcbd8f56f725f829d5af6054a6e0c
Ports: 23000/TCP, 22999/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/.supervisor/workspacekit
ring0
State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 13 Sep 2022 05:38:47 -0400
Finished: Tue, 13 Sep 2022 05:40:47 -0400
Ready: False
Restart Count: 0
Limits:
cpu: 8
ephemeral-storage: 20Gi
memory: 16Gi
Requests:
cpu: 2
ephemeral-storage: 10Gi
memory: 12Gi
Readiness: http-get http://:22999/_supervisor/v1/status/content/wait/true delay=2s timeout=1s period=1s #success=1 #failure=600
Mounts:
/.workspace from daemon-mount (rw)
/workspace from vol-this-workspace (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
vol-this-workspace:
Type: HostPath (bare host directory volume)
Path: /var/gitpod/workspaces/64bdc59e-d708-4304-83ec-0ed5fa6c9797
HostPathType: DirectoryOrCreate
daemon-mount:
Type: HostPath (bare host directory volume)
Path: /var/gitpod/workspaces/64bdc59e-d708-4304-83ec-0ed5fa6c9797-daemon
HostPathType: DirectoryOrCreate
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoExecute op=Exists
node.kubernetes.io/memory-pressure:NoExecute op=Exists
node.kubernetes.io/network-unavailable:NoExecute op=Exists for 30s
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m kumquat Successfully assigned default/ws-64bdc59e-d708-4304-83ec-0ed5fa6c9797 to workspace-ws-us64-internal-xl-pool-9xqw
Normal Pulling 13m kubelet Pulling image "reg.ws-us64.gitpod.io:20000/remote/64bdc59e-d708-4304-83ec-0ed5fa6c9797"
Normal Pulled 13m kubelet Successfully pulled image "reg.ws-us64.gitpod.io:20000/remote/64bdc59e-d708-4304-83ec-0ed5fa6c9797" in 4.390105086s
Normal Created 13m kubelet Created container workspace
Normal Started 13m kubelet Started container workspace
Warning Unhealthy 12m (x21 over 13m) kubelet Readiness probe failed: Get "http://10.20.255.224:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.20.255.224:22999: connect: connection refusedSteps to reproduce
Unsure
Workspace affected
gitpodio-templatepython-vlhvnj7u60f
Expected behavior
Ideally it would connect, and the workspace would start.
Example repository
No response
