Closed
Description
Bug description
Everything looks fine, I can register with our self hosted GitLab, but when I start creating the first workspace from GitLab I get
Connecting to workspace logs...
Error: image build did not produce a workspace image
Steps to reproduce
- Installed with installer TAG sje-installer-custom-ingress.2 see [installer]: grant option for disabling proxy load balancer #7280
- K3S v1.21.7+k3s1
- Flannel Networking on private subnet
- No Traefik
- Installed NGINX Ingress via Helm and with Policies mentioned in [installer]: grant option for disabling proxy load balancer #7280
- Config:
apiVersion: v1
authProviders: []
blockNewUsers:
enabled: false
passlist: []
certificate:
kind: secret
name: https-certificates
containerRegistry:
inCluster: true
database:
inCluster: true
domain: my.domain.com
imagePullSecrets: null
jaegerOperator:
inCluster: true
kind: Full
meta:
ingress:
serviceType: ClusterIP
metadata:
region: local
objectStorage:
inCluster: true
observability:
logLevel: info
repository: eu.gcr.io/gitpod-core-dev/build
workspace:
resources:
requests:
cpu: "1"
memory: 2Gi
runtime:
containerdRuntimeDir: /var/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io
containerdSocket: /var/run/k3s/containerd/containerd.sock
fsShiftMethod: fuse
What I get from image-builder-mk3 when the process is starting:
{"host":"registry.my.domain.com","level":"info","message":"trying next host - response was http.StatusNotFound","severity":"INFO","time":"2021-12-21T06:32:06Z"}
{"host":"registry.my.domain.com","level":"info","message":"trying next host - response was http.StatusNotFound","severity":"INFO","time":"2021-12-21T06:32:06Z"}
{"host":"registry.my.domain.com","level":"info","message":"trying next host - response was http.StatusNotFound","severity":"INFO","time":"2021-12-21T06:32:22Z"}
But name resolution inside container is working and gives me the external IP:
kubectl exec -it image-builder-mk3-599d9b9678-5n7wq image-builder-mk3 -- nslookup registry.my.domain.com
Server: 10.43.0.10
Address: 10.43.0.10:53
registry.my.domain.com canonical name = my.domain.com
registry.my.domain.com canonical name = my.domain.com
Name: my.domain.com
Address: 95.xxx.xxx.xxx
Registry is working from the outside:
❯ curl -I https://registry.my.domain.com/v2
HTTP/2 401
date: Tue, 21 Dec 2021 07:04:25 GMT
www-authenticate: Basic realm="Docker Registry"
strict-transport-security: max-age=15724800; includeSubDomains
There is no pod image-build* coming up.
Logs:
ws-daemon-954pl kube-rbac-proxy I1221 06:21:31.287673 10290 main.go:379] Listening insecurely on [10.42.1.18]:9500
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"ERROR","time":"2021-12-21T06:46:00.424Z","message":"Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778.","loggedViaConsole":true}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:04Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:04Z"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"ERROR","time":"2021-12-21T06:46:06.540Z","message":"Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778.","loggedViaConsole":true}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:09Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:09Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:14Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:14Z"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"component":"ws-manager-bridge","severity":"INFO","time":"2021-12-21T06:46:16.340Z","message":"reconciling clusters...","payload":{"allClusters":{}}}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"component":"ws-manager-bridge","severity":"INFO","time":"2021-12-21T06:46:16.340Z","message":"done reconciling.","payload":{"allClusters":{}}}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:19Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:19Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:24Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:24Z"}
ws-manager-c8d6d4767-d28nf ws-manager {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"starting new workspace","serviceContext":{"service":"ws-manager","version":"commit-1971c9cbae1bf2b4378c6d6c57a7d251a4063f21"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"1d60d187-6d80-4d87-940e-82f94819b481","InstanceID":"1d60d187-6d80-4d87-940e-82f94819b481","URL":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"","Ports":null,"Auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"},"StartedAt":"2021-12-21T06:46:26Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","workspace":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"attempt":1,"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"scheduling pod","name":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceType":"imagebuild"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"attempt":1,"ghostToDelete":"","instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"ghostToDelete","name":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","unschedulable":false,"userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceType":"imagebuild"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"counts":{"Regular":0,"Ghost":0},"level":"debug","message":"status update","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"1d60d187-6d80-4d87-940e-82f94819b481","InstanceID":"1d60d187-6d80-4d87-940e-82f94819b481","URL":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"","Ports":null,"Auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"},"StartedAt":"2021-12-21T06:46:26Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","workspace":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"WARNING","time":"2021-12-21T06:46:26.746Z","context":{"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","userId":"image-builder"},"message":"Received a status update for an unknown instance","payload":{"status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","statusVersion":107483574173696,"metadata":{"owner":"image-builder","metaId":"1d60d187-6d80-4d87-940e-82f94819b481","startedAt":{"seconds":1640069186,"nanos":0},"annotationsMap":[["baseref","docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8"],["ref","registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"]]},"spec":{"workspaceImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecatedIdeImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","exposedPortsList":[],"type":4,"timeout":"1h0m0s","ideImage":{"webRef":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","desktopRef":"","supervisorRef":""}},"phase":1,"conditions":{"failed":"","timeout":"","pullingImages":0,"snapshot":"","finalBackupComplete":0,"deployed":0,"networkNotReady":0,"headlessTaskFailed":"","stoppedByRequest":0},"message":"pod is pending","runtime":{"nodeName":"","podName":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","nodeIp":""},"auth":{"admission":0,"ownerToken":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}}}}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"attempt":1,"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"bound to node: gitpod-ws1","name":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceType":"imagebuild"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"1d60d187-6d80-4d87-940e-82f94819b481","InstanceID":"1d60d187-6d80-4d87-940e-82f94819b481","URL":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"","Ports":null,"Auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"},"StartedAt":"2021-12-21T06:46:26Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","workspace":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"WARNING","time":"2021-12-21T06:46:26.763Z","context":{"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","userId":"image-builder"},"message":"Received a status update for an unknown instance","payload":{"status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","statusVersion":107483574173697,"metadata":{"owner":"image-builder","metaId":"1d60d187-6d80-4d87-940e-82f94819b481","startedAt":{"seconds":1640069186,"nanos":0},"annotationsMap":[["baseref","docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8"],["ref","registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"]]},"spec":{"workspaceImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecatedIdeImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","exposedPortsList":[],"type":4,"timeout":"1h0m0s","ideImage":{"webRef":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","desktopRef":"","supervisorRef":""}},"phase":1,"conditions":{"failed":"","timeout":"","pullingImages":0,"snapshot":"","finalBackupComplete":0,"deployed":0,"networkNotReady":0,"headlessTaskFailed":"","stoppedByRequest":0},"message":"pod is pending","runtime":{"nodeName":"gitpod-ws1","podName":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","nodeIp":""},"auth":{"admission":0,"ownerToken":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}}}}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"InitWorkspace called","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"hooks":4,"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"running lifecycle hooks","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","state":"initializing","time":"2021-12-21T06:46:26Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"established IWS server","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"WARNING","time":"2021-12-21T06:46:26.790Z","context":{"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","userId":"image-builder"},"message":"Received a status update for an unknown instance","payload":{"status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","statusVersion":107483574173698,"metadata":{"owner":"image-builder","metaId":"1d60d187-6d80-4d87-940e-82f94819b481","startedAt":{"seconds":1640069186,"nanos":0},"annotationsMap":[["baseref","docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8"],["ref","registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"]]},"spec":{"workspaceImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecatedIdeImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","exposedPortsList":[],"type":4,"timeout":"1h0m0s","ideImage":{"webRef":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","desktopRef":"","supervisorRef":""}},"phase":2,"conditions":{"failed":"","timeout":"","pullingImages":1,"snapshot":"","finalBackupComplete":0,"deployed":0,"networkNotReady":0,"headlessTaskFailed":"","stoppedByRequest":0},"message":"containers are being created","runtime":{"nodeName":"gitpod-ws1","podName":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","nodeIp":"192.168.72.2"},"auth":{"admission":0,"ownerToken":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}}}}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/cgroups/file.go:87","func":"github.com/opencontainers/runc/libcontainer/cgroups.prepareOpenat2.func1","level":"debug","msg":"openat2 not available, falling back to securejoin","time":"2021-12-21T06:46:26Z"}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"hit route handler","portID":"","routeHandler":"HandleDirectSupervisorRoute (authenticated: true)","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:26Z","url":"/_supervisor/v1/status/tasks","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"proxy-passing request","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","targetURL":"http://:22999","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec[4570]: =\u003e nsexec container setup","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: ~\u003e nsexec stage-0","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: spawn stage-1","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: -\u003e stage-1 synchronisation loop","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[4573]: ~\u003e nsexec stage-1","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[4573]: unshare remaining namespace (except cgroupns)","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[4573]: spawn stage-2","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[4573]: request stage-0 to forward stage-2 pid (4574)","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-2[1]: ~\u003e nsexec stage-2","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: stage-1 requested pid to be forwarded","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: forward stage-1 (4573) and stage-2 (4574) pids to runc","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[4573]: signal completion to stage-0","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-1[4573]: \u003c~ nsexec stage-1","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: stage-1 complete","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: \u003c- stage-1 synchronisation loop","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: -\u003e stage-2 synchronisation loop","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: signalling stage-2 to run","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-2[1]: signal completion to stage-0","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-2[1]: \u003c= nsexec container setup","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-2[1]: booting up go runtime ...","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: stage-2 complete","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: \u003c- stage-2 synchronisation loop","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"nsexec-0[4570]: \u003c~ nsexec stage-0","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"child process in init()","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/libcontainer/logs/logs.go:69","func":"github.com/opencontainers/runc/libcontainer/logs.processEntry","level":"debug","msg":"init: closing the pipe to signal completion","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/signals.go:104","func":"main.(*signalHandler).forward","level":"debug","msg":"sending signal to process urgent I/O condition","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/signals.go:104","func":"main.(*signalHandler).forward","level":"debug","msg":"sending signal to process urgent I/O condition","time":"2021-12-21T06:46:26Z"}
ws-daemon-sgg58 ws-daemon {"file":"github.com/opencontainers/runc/signals.go:94","func":"main.(*signalHandler).forward","level":"debug","msg":"process exited","pid":4574,"status":0,"time":"2021-12-21T06:46:27Z"}
ws-daemon-sgg58 ws-daemon {"hooks":3,"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"running lifecycle hooks","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","state":"ready","time":"2021-12-21T06:46:27Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"found sandbox - adding to label cache","podname":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:27Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"hit route handler","portID":"","routeHandler":"HandleDirectSupervisorRoute (authenticated: true)","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:27Z","url":"/_supervisor/v1/status/tasks","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"proxy-passing request","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","targetURL":"http://:22999","time":"2021-12-21T06:46:27Z"}
ws-daemon-sgg58 ws-daemon {"ID":"613bccbd69442e30cd6394d1c5c8422ccd4fdf94841a8260452e0eefe51525a1","instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"found workspace container - updating label cache","podname":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"cid":"613bccbd69442e30cd6394d1c5c8422ccd4fdf94841a8260452e0eefe51525a1","instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"found task","rootfs":"","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","upperdir":"","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"container":"613bccbd69442e30cd6394d1c5c8422ccd4fdf94841a8260452e0eefe51525a1","instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"dispatch found new workspace container","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"containerID":"613bccbd69442e30cd6394d1c5c8422ccd4fdf94841a8260452e0eefe51525a1","instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"started new resource governer","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"level":"debug","message":"FSShift","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","type":1}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"nsexec started","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"join mnt namespace: 5","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"chroot: 3","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"chcwd: 4","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"nsexec started","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"join mnt namespace: 5","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"chroot: 3","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"chcwd: 4","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"nsexec started","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"join mnt namespace: 5","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"chroot: 3","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"chcwd: 4","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"configuredShift":1,"fwb":false,"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"info","message":"fs-shift using fuse","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"INFO","time":"2021-12-21T06:46:28Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"nsexec started","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"join mnt namespace: 5","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"chroot: 3","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"chcwd: 4","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:28Z","userId":"","workspaceId":""}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:29Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:29Z"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"1d60d187-6d80-4d87-940e-82f94819b481","InstanceID":"1d60d187-6d80-4d87-940e-82f94819b481","URL":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.25","Ports":null,"Auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"},"StartedAt":"2021-12-21T06:46:26Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","workspace":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"WARNING","time":"2021-12-21T06:46:29.295Z","context":{"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","userId":"image-builder"},"message":"Received a status update for an unknown instance","payload":{"status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","statusVersion":107483574370304,"metadata":{"owner":"image-builder","metaId":"1d60d187-6d80-4d87-940e-82f94819b481","startedAt":{"seconds":1640069186,"nanos":0},"annotationsMap":[["baseref","docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8"],["ref","registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"]]},"spec":{"workspaceImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecatedIdeImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","exposedPortsList":[],"type":4,"timeout":"1h0m0s","ideImage":{"webRef":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","desktopRef":"","supervisorRef":""}},"phase":4,"conditions":{"failed":"container workspace ran with an error: exit code 1","timeout":"","pullingImages":0,"snapshot":"","finalBackupComplete":0,"deployed":0,"networkNotReady":0,"headlessTaskFailed":"","stoppedByRequest":0},"message":"","runtime":{"nodeName":"gitpod-ws1","podName":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","nodeIp":"192.168.72.2"},"auth":{"admission":0,"ownerToken":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}}}}
ws-manager-c8d6d4767-d28nf ws-manager {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"error","message":"workspace failed","serviceContext":{"service":"ws-manager","version":"commit-1971c9cbae1bf2b4378c6d6c57a7d251a4063f21"},"severity":"ERROR","status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","status_version":107483574370304,"metadata":{"owner":"image-builder","meta_id":"1d60d187-6d80-4d87-940e-82f94819b481","started_at":{"seconds":1640069186},"annotations":{"baseref":"docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8","ref":"registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"}},"spec":{"workspace_image":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecated_ide_image":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","type":4,"timeout":"1h0m0s","ide_image":{"web_ref":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44"}},"phase":4,"conditions":{"failed":"container workspace ran with an error: exit code 1"},"runtime":{"node_name":"gitpod-ws1","pod_name":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","node_ip":"192.168.72.2"},"auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}},"time":"2021-12-21T06:46:29Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"1d60d187-6d80-4d87-940e-82f94819b481","InstanceID":"1d60d187-6d80-4d87-940e-82f94819b481","URL":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.25","Ports":null,"Auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"},"StartedAt":"2021-12-21T06:46:26Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","workspace":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"1d60d187-6d80-4d87-940e-82f94819b481","InstanceID":"1d60d187-6d80-4d87-940e-82f94819b481","URL":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.25","Ports":null,"Auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"},"StartedAt":"2021-12-21T06:46:26Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","workspace":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481"}
ws-manager-c8d6d4767-d28nf ws-manager {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"error","message":"workspace failed","serviceContext":{"service":"ws-manager","version":"commit-1971c9cbae1bf2b4378c6d6c57a7d251a4063f21"},"severity":"ERROR","status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","status_version":107483574370306,"metadata":{"owner":"image-builder","meta_id":"1d60d187-6d80-4d87-940e-82f94819b481","started_at":{"seconds":1640069186},"annotations":{"baseref":"docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8","ref":"registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"}},"spec":{"workspace_image":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecated_ide_image":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","type":4,"timeout":"1h0m0s","ide_image":{"web_ref":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44"}},"phase":4,"conditions":{"failed":"container workspace ran with an error: exit code 1"},"runtime":{"node_name":"gitpod-ws1","pod_name":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","node_ip":"192.168.72.2"},"auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}},"time":"2021-12-21T06:46:29Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"WARNING","time":"2021-12-21T06:46:29.336Z","context":{"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","userId":"image-builder"},"message":"Received a status update for an unknown instance","payload":{"status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","statusVersion":107483574370306,"metadata":{"owner":"image-builder","metaId":"1d60d187-6d80-4d87-940e-82f94819b481","startedAt":{"seconds":1640069186,"nanos":0},"annotationsMap":[["baseref","docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8"],["ref","registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"]]},"spec":{"workspaceImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecatedIdeImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","exposedPortsList":[],"type":4,"timeout":"1h0m0s","ideImage":{"webRef":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","desktopRef":"","supervisorRef":""}},"phase":4,"conditions":{"failed":"container workspace ran with an error: exit code 1","timeout":"","pullingImages":0,"snapshot":"","finalBackupComplete":0,"deployed":0,"networkNotReady":0,"headlessTaskFailed":"","stoppedByRequest":0},"message":"","runtime":{"nodeName":"gitpod-ws1","podName":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","nodeIp":"192.168.72.2"},"auth":{"admission":0,"ownerToken":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}}}}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"DisposeWorkspace called","req":"id:\"1d60d187-6d80-4d87-940e-82f94819b481\"","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","userId":"","workspaceId":""}
ws-daemon-sgg58 ws-daemon {"hooks":0,"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"running lifecycle hooks","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","state":"disposing","time":"2021-12-21T06:46:29Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","loc":"/mnt/workingarea/1d60d187-6d80-4d87-940e-82f94819b481","message":"did not find a Git working copy - not updating Git status","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"hooks":2,"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"running lifecycle hooks","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","state":"disposed","time":"2021-12-21T06:46:29Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"stopped IWS server","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"WARNING","time":"2021-12-21T06:46:29.351Z","context":{"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","userId":"image-builder"},"message":"Received a status update for an unknown instance","payload":{"status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","statusVersion":107483574370308,"metadata":{"owner":"image-builder","metaId":"1d60d187-6d80-4d87-940e-82f94819b481","startedAt":{"seconds":1640069186,"nanos":0},"annotationsMap":[["baseref","docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8"],["ref","registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"]]},"spec":{"workspaceImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecatedIdeImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","exposedPortsList":[],"type":4,"timeout":"1h0m0s","ideImage":{"webRef":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","desktopRef":"","supervisorRef":""}},"phase":5,"conditions":{"failed":"","timeout":"","pullingImages":0,"snapshot":"","finalBackupComplete":0,"deployed":0,"networkNotReady":0,"headlessTaskFailed":"","stoppedByRequest":0},"message":"","runtime":{"nodeName":"gitpod-ws1","podName":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","nodeIp":"192.168.72.2"},"auth":{"admission":0,"ownerToken":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}}}}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"1d60d187-6d80-4d87-940e-82f94819b481","InstanceID":"1d60d187-6d80-4d87-940e-82f94819b481","URL":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.25","Ports":null,"Auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"},"StartedAt":"2021-12-21T06:46:26Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","workspace":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"counts":{"Regular":0,"Ghost":0},"level":"debug","message":"status update","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"WARNING","time":"2021-12-21T06:46:29.382Z","context":{"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","userId":"image-builder"},"message":"Received a status update for an unknown instance","payload":{"status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","statusVersion":107483574370309,"metadata":{"owner":"image-builder","metaId":"1d60d187-6d80-4d87-940e-82f94819b481","startedAt":{"seconds":1640069186,"nanos":0},"annotationsMap":[["baseref","docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8"],["ref","registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"]]},"spec":{"workspaceImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecatedIdeImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","exposedPortsList":[],"type":4,"timeout":"1h0m0s","ideImage":{"webRef":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","desktopRef":"","supervisorRef":""}},"phase":6,"conditions":{"failed":"","timeout":"","pullingImages":0,"snapshot":"","finalBackupComplete":1,"deployed":0,"networkNotReady":0,"headlessTaskFailed":"","stoppedByRequest":0},"message":"","runtime":{"nodeName":"gitpod-ws1","podName":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","nodeIp":"192.168.72.2"},"auth":{"admission":0,"ownerToken":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}}}}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"1d60d187-6d80-4d87-940e-82f94819b481","InstanceID":"1d60d187-6d80-4d87-940e-82f94819b481","URL":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.25","Ports":null,"Auth":{"owner_token":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"},"StartedAt":"2021-12-21T06:46:26Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","workspace":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"WARNING","time":"2021-12-21T06:46:29.413Z","context":{"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481","userId":"image-builder"},"message":"Received a status update for an unknown instance","payload":{"status":{"id":"1d60d187-6d80-4d87-940e-82f94819b481","statusVersion":107483574370310,"metadata":{"owner":"image-builder","metaId":"1d60d187-6d80-4d87-940e-82f94819b481","startedAt":{"seconds":1640069186,"nanos":0},"annotationsMap":[["baseref","docker.io/gitpod/workspace-full@sha256:46859378f3067e50e8d9509e270f67a0447bb28f5d2a2f0c723dc916411a46c8"],["ref","registry.my.domain.com/workspace-images:e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77"]]},"spec":{"workspaceImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","deprecatedIdeImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","headless":true,"url":"https://1d60d187-6d80-4d87-940e-82f94819b481.ws.my.domain.com","exposedPortsList":[],"type":4,"timeout":"1h0m0s","ideImage":{"webRef":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","desktopRef":"","supervisorRef":""}},"phase":6,"conditions":{"failed":"","timeout":"","pullingImages":0,"snapshot":"","finalBackupComplete":1,"deployed":0,"networkNotReady":0,"headlessTaskFailed":"","stoppedByRequest":0},"message":"","runtime":{"nodeName":"gitpod-ws1","podName":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481","nodeIp":"192.168.72.2"},"auth":{"admission":0,"ownerToken":"cnv8YBCsLskBzTZa2vp.DlvlubQQKHpt"}}}}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"removing workspace from store","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","workspace":"imagebuild-1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-sgg58 ws-daemon {"containerID":"613bccbd69442e30cd6394d1c5c8422ccd4fdf94841a8260452e0eefe51525a1","instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"resource controller shutting down","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","userId":"","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-954pl ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"received pod deletion for a workspace, but have not seen it before. Probably another node. Ignoring update.","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-lpglm ws-daemon {"instanceId":"1d60d187-6d80-4d87-940e-82f94819b481","level":"debug","message":"received pod deletion for a workspace, but have not seen it before. Probably another node. Ignoring update.","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:29Z","userId":"image-builder","workspaceId":"1d60d187-6d80-4d87-940e-82f94819b481"}
ws-daemon-954pl ws-daemon {"bvail":226545401856,"level":"debug","message":"checked for available disk space","minBytesAvail":21474836480,"serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:31Z"}
ws-daemon-954pl ws-daemon {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"nodes \"gitpod-ws2\" is forbidden: User \"system:serviceaccount:gitpod:ws-daemon\" cannot get resource \"nodes\" in API group \"\" at the cluster scope","level":"error","message":"cannot update node label","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"ERROR","time":"2021-12-21T06:46:31Z"}
ws-daemon-sgg58 ws-daemon {"bvail":225901314048,"level":"debug","message":"checked for available disk space","minBytesAvail":21474836480,"serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:31Z"}
ws-daemon-sgg58 ws-daemon {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"nodes \"gitpod-ws1\" is forbidden: User \"system:serviceaccount:gitpod:ws-daemon\" cannot get resource \"nodes\" in API group \"\" at the cluster scope","level":"error","message":"cannot update node label","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"ERROR","time":"2021-12-21T06:46:31Z"}
ws-daemon-lpglm ws-daemon {"bvail":226779369472,"level":"debug","message":"checked for available disk space","minBytesAvail":21474836480,"serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","time":"2021-12-21T06:46:31Z"}
ws-daemon-lpglm ws-daemon {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"nodes \"gitpod-ws3\" is forbidden: User \"system:serviceaccount:gitpod:ws-daemon\" cannot get resource \"nodes\" in API group \"\" at the cluster scope","level":"error","message":"cannot update node label","serviceContext":{"service":"ws-daemon","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"ERROR","time":"2021-12-21T06:46:31Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:34Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:34Z"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"ERROR","time":"2021-12-21T06:46:39.212Z","message":"Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778.","loggedViaConsole":true}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:39Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:39Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:44Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:44Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:49Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:49Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:54Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:54Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:46:59Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:46:59Z"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"ERROR","time":"2021-12-21T06:47:00.424Z","message":"Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778.","loggedViaConsole":true}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:47:04Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:47:04Z"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"ws-manager-bridge","version":"<ts-not-set>"},"component":"ws-manager-bridge","severity":"ERROR","time":"2021-12-21T06:47:06.540Z","message":"Error in fetching sampling strategy: Error: connect ECONNREFUSED 0.0.0.0:5778.","loggedViaConsole":true}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:47:09Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:47:09Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:47:14Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:47:14Z"}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"component":"ws-manager-bridge","severity":"INFO","time":"2021-12-21T06:47:16.343Z","message":"reconciling clusters...","payload":{"allClusters":{}}}
ws-manager-bridge-dc7f8cf6b-rlmzf ws-manager-bridge {"component":"ws-manager-bridge","severity":"INFO","time":"2021-12-21T06:47:16.343Z","message":"done reconciling.","payload":{"allClusters":{}}}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"level":"warning","limit":0,"message":"max number of ghost workspace reached","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"severity":"WARNING","time":"2021-12-21T06:47:19Z"}
ws-scheduler-7d988846db-6mwf5 ws-scheduler {"delta":80,"level":"debug","message":"(de)scheduled ghost workspaces","serviceContext":{"service":"ws-scheduler","version":"commit-b577c7746c3d0b5b90c7b7e8864c8ebb1e82c4a3"},"setpoint":80,"severity":"DEBUG","time":"2021-12-21T06:47:19Z"}
server-6598d677f4-vjjqq server {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"server","version":"sje-installer-custom-ingress.2"},"component":"server","severity":"WARNING","time":"2021-12-21T06:46:29.376Z","context":{"instanceId":"9dbce937-fdc6-4561-a083-a1f036807e4f","userId":"7cb684a4-5869-441e-9290-426b6d5c8a46","workspaceId":"purple-hummingbird-rhcrl68o"},"message":"workspace image build failed: image build did not produce a workspace image"}
server-6598d677f4-vjjqq server {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","serviceContext":{"service":"server","version":"sje-installer-custom-ingress.2"},"stack_trace":"Error: image build did not produce a workspace image\n at WorkspaceStarterEE.<anonymous> (/app/node_modules/@gitpod/server/dist/src/workspace/workspace-starter.js:496:31)\n at Generator.next (<anonymous>)\n at fulfilled (/app/node_modules/@gitpod/server/dist/src/workspace/workspace-starter.js:19:58)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)","component":"server","severity":"ERROR","time":"2021-12-21T06:46:29.425Z","message":"error starting instance","error":"Error: image build did not produce a workspace image\n at WorkspaceStarterEE.<anonymous> (/app/node_modules/@gitpod/server/dist/src/workspace/workspace-starter.js:496:31)\n at Generator.next (<anonymous>)\n at fulfilled (/app/node_modules/@gitpod/server/dist/src/workspace/workspace-starter.js:19:58)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)","payload":{"instanceId":"9dbce937-fdc6-4561-a083-a1f036807e4f"}}
proxy-7b695dfbb7-tjtpc proxy {"severity":"ERROR","timestamp":"2021-12-21T07:02:34.479886393Z","logName":"http.log.access.log2","httpRequest":{"requestMethod":"HEAD","protocol":"HTTP/1.1","status":401,"responseSize":0,"userAgent":"containerd/1.5.5+unknown","requestUrl":"/v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77","requestHost":"registry.my.domain.com"}}
proxy-7b695dfbb7-tjtpc proxy {"severity":"ERROR","timestamp":"2021-12-21T07:02:34.487936005Z","logName":"http.log.access.log2","httpRequest":{"requestMethod":"HEAD","protocol":"HTTP/1.1","status":404,"responseSize":0,"userAgent":"containerd/1.5.5+unknown","requestUrl":"/v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77","requestHost":"registry.my.domain.com"}}
proxy-7b695dfbb7-tjtpc proxy {"severity":"ERROR","timestamp":"2021-12-21T07:02:34.502207906Z","logName":"http.log.access.log2","httpRequest":{"requestMethod":"HEAD","protocol":"HTTP/1.1","status":401,"responseSize":0,"userAgent":"containerd/1.5.5+unknown","requestUrl":"/v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77","requestHost":"registry.my.domain.com"}}
proxy-7b695dfbb7-tjtpc proxy {"severity":"ERROR","timestamp":"2021-12-21T07:02:34.509957746Z","logName":"http.log.access.log2","httpRequest":{"requestMethod":"HEAD","protocol":"HTTP/1.1","status":404,"responseSize":0,"userAgent":"containerd/1.5.5+unknown","requestUrl":"/v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77","requestHost":"registry.my.domain.com"}}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","InstanceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","URL":"https://222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"","Ports":null,"Auth":{"owner_token":"Sd3TDjp.xKtKFRL2SiRPNZeF1HDOHw8G"},"StartedAt":"2021-12-21T07:02:34Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:34Z","workspace":"imagebuild-222c6c3c-2988-400e-9cbc-30dd6557e270"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","InstanceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","URL":"https://222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"","Ports":null,"Auth":{"owner_token":"Sd3TDjp.xKtKFRL2SiRPNZeF1HDOHw8G"},"StartedAt":"2021-12-21T07:02:34Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:34Z","workspace":"imagebuild-222c6c3c-2988-400e-9cbc-30dd6557e270"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","InstanceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","URL":"https://222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"","Ports":null,"Auth":{"owner_token":"Sd3TDjp.xKtKFRL2SiRPNZeF1HDOHw8G"},"StartedAt":"2021-12-21T07:02:34Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:34Z","workspace":"imagebuild-222c6c3c-2988-400e-9cbc-30dd6557e270"}
proxy-7b695dfbb7-tjtpc proxy {"severity":"ERROR","timestamp":"2021-12-21T07:02:34.649543614Z","logName":"http.log.access.log1","httpRequest":{"requestMethod":"GET","protocol":"HTTP/1.1","status":502,"responseSize":0,"userAgent":"Go-http-client/1.1","requestUrl":"/_supervisor/v1/status/tasks","requestHost":"222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com"}}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"hit route handler","portID":"","routeHandler":"HandleDirectSupervisorRoute (authenticated: true)","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:34Z","url":"/_supervisor/v1/status/tasks","workspaceId":"222c6c3c-2988-400e-9cbc-30dd6557e270"}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"proxy-passing request","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","targetURL":"http://:22999","time":"2021-12-21T07:02:34Z"}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"hit route handler","portID":"","routeHandler":"HandleDirectSupervisorRoute (authenticated: true)","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:35Z","url":"/_supervisor/v1/status/tasks","workspaceId":"222c6c3c-2988-400e-9cbc-30dd6557e270"}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"proxy-passing request","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","targetURL":"http://:22999","time":"2021-12-21T07:02:35Z"}
proxy-7b695dfbb7-tjtpc proxy {"severity":"ERROR","timestamp":"2021-12-21T07:02:35.656148278Z","logName":"http.log.access.log1","httpRequest":{"requestMethod":"GET","protocol":"HTTP/1.1","status":502,"responseSize":0,"userAgent":"Go-http-client/1.1","requestUrl":"/_supervisor/v1/status/tasks","requestHost":"222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com"}}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","InstanceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","URL":"https://222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.26","Ports":null,"Auth":{"owner_token":"Sd3TDjp.xKtKFRL2SiRPNZeF1HDOHw8G"},"StartedAt":"2021-12-21T07:02:34Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:37Z","workspace":"imagebuild-222c6c3c-2988-400e-9cbc-30dd6557e270"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","InstanceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","URL":"https://222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.26","Ports":null,"Auth":{"owner_token":"Sd3TDjp.xKtKFRL2SiRPNZeF1HDOHw8G"},"StartedAt":"2021-12-21T07:02:34Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:37Z","workspace":"imagebuild-222c6c3c-2988-400e-9cbc-30dd6557e270"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","InstanceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","URL":"https://222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.26","Ports":null,"Auth":{"owner_token":"Sd3TDjp.xKtKFRL2SiRPNZeF1HDOHw8G"},"StartedAt":"2021-12-21T07:02:34Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:37Z","workspace":"imagebuild-222c6c3c-2988-400e-9cbc-30dd6557e270"}
proxy-7b695dfbb7-tjtpc proxy {"severity":"ERROR","timestamp":"2021-12-21T07:02:37.097891362Z","logName":"http.log.access.log2","httpRequest":{"requestMethod":"HEAD","protocol":"HTTP/1.1","status":401,"responseSize":0,"userAgent":"containerd/1.5.5+unknown","requestUrl":"/v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77","requestHost":"registry.my.domain.com"}}
proxy-7b695dfbb7-tjtpc proxy {"severity":"ERROR","timestamp":"2021-12-21T07:02:37.106595152Z","logName":"http.log.access.log2","httpRequest":{"requestMethod":"HEAD","protocol":"HTTP/1.1","status":404,"responseSize":0,"userAgent":"containerd/1.5.5+unknown","requestUrl":"/v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77","requestHost":"registry.my.domain.com"}}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","InstanceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","URL":"https://222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.26","Ports":null,"Auth":{"owner_token":"Sd3TDjp.xKtKFRL2SiRPNZeF1HDOHw8G"},"StartedAt":"2021-12-21T07:02:34Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:37Z","workspace":"imagebuild-222c6c3c-2988-400e-9cbc-30dd6557e270"}
ws-proxy-7575949d86-pk7gj ws-proxy {"details":{"WorkspaceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","InstanceID":"222c6c3c-2988-400e-9cbc-30dd6557e270","URL":"https://222c6c3c-2988-400e-9cbc-30dd6557e270.ws.my.domain.com","IDEImage":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","SupervisorImage":"","IDEPublicPort":"443","IPAddress":"10.42.0.26","Ports":null,"Auth":{"owner_token":"Sd3TDjp.xKtKFRL2SiRPNZeF1HDOHw8G"},"StartedAt":"2021-12-21T07:02:34Z"},"level":"debug","message":"adding/updating workspace details","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:37Z","workspace":"imagebuild-222c6c3c-2988-400e-9cbc-30dd6557e270"}
ws-proxy-7575949d86-pk7gj ws-proxy {"level":"debug","message":"removing workspace from store","serviceContext":{"service":"ws-proxy","version":"commit-30969a22c9426815874744194d80d17eaac807ec"},"severity":"DEBUG","time":"2021-12-21T07:02:37Z","workspace":"imagebuild-222c6c3c-2988-400e-9cbc-30dd6557e270"}
registry-6bd464448c-xv588 docker-registry 10.42.3.30 - - [21/Dec/2021:07:05:34 +0000] "HEAD /v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77 HTTP/2.0" 404 154 "" "containerd/1.5.5+unknown"
registry-6bd464448c-xv588 docker-registry time="2021-12-21T07:05:34.473114004Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77" err.message="manifest unknown" go.version=go1.11.2 http.request.host=registry.my.domain.com http.request.id=1f2d3b3b-9127-473c-86b0-ff35f664a53f http.request.method=HEAD http.request.remoteaddr=10.42.2.2 http.request.uri="/v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77" http.request.useragent="containerd/1.5.5+unknown" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.281948ms http.response.status=404 http.response.written=154 vars.name=workspace-images vars.reference=e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77
registry-6bd464448c-xv588 docker-registry 10.42.3.30 - - [21/Dec/2021:07:05:34 +0000] "HEAD /v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77 HTTP/2.0" 404 154 "" "containerd/1.5.5+unknown"
registry-6bd464448c-xv588 docker-registry time="2021-12-21T07:05:34.48862024Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77" err.message="manifest unknown" go.version=go1.11.2 http.request.host=registry.my.domain.com http.request.id=214b772a-299d-4efa-8fb5-55de89b32c93 http.request.method=HEAD http.request.remoteaddr=10.42.2.2 http.request.uri="/v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77" http.request.useragent="containerd/1.5.5+unknown" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.761178ms http.response.status=404 http.response.written=154 vars.name=workspace-images vars.reference=e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77
registry-facade-f26zv registry-facade {"instanceId":"51a12ba3-9d99-4e5a-8ff0-18b7a931b3c6","level":"debug","message":"get manifest","serviceContext":{"service":"registry-facade","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","spec":{"base_ref":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","ide_ref":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44"},"tag":"latest","time":"2021-12-21T07:05:35Z","userId":"","workspaceId":""}
registry-facade-f26zv registry-facade {"host":"eu.gcr.io","level":"debug","message":"resolving","severity":"DEBUG","time":"2021-12-21T07:05:35Z"}
registry-facade-f26zv registry-facade {"host":"eu.gcr.io","level":"debug","message":"do request","request.header.accept":"application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*","request.header.user-agent":"containerd/1.5.5+unknown","request.method":"HEAD","severity":"DEBUG","time":"2021-12-21T07:05:35Z","url":"https://eu.gcr.io/v2/gitpod-core-dev/build/image-builder-mk3/bob/manifests/075a12d448c9205dd995e1acff4df74f46569b44"}
registry-facade-f26zv registry-facade {"host":"eu.gcr.io","level":"debug","message":"fetch response received","response.header.alt-svc":"h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"","response.header.content-length":"2836","response.header.content-type":"application/vnd.docker.distribution.manifest.v2+json","response.header.date":"Tue, 21 Dec 2021 07:05:35 GMT","response.header.docker-content-digest":"sha256:9fa7853bccab540cd7f0af1ee6e43d64fb7d061fa17983006fe5dadfb82b211f","response.header.docker-distribution-api-version":"registry/2.0","response.header.server":"Docker Registry","response.header.x-frame-options":"SAMEORIGIN","response.header.x-xss-protection":"0","response.status":"200 OK","severity":"DEBUG","time":"2021-12-21T07:05:35Z","url":"https://eu.gcr.io/v2/gitpod-core-dev/build/image-builder-mk3/bob/manifests/075a12d448c9205dd995e1acff4df74f46569b44"}
registry-facade-f26zv registry-facade {"desc.digest":"sha256:9fa7853bccab540cd7f0af1ee6e43d64fb7d061fa17983006fe5dadfb82b211f","host":"eu.gcr.io","level":"debug","message":"resolved","severity":"DEBUG","time":"2021-12-21T07:05:35Z"}
registry-facade-f26zv registry-facade {"digest":"sha256:c6403c89f512020e7e2c34422eb19220f9d61c9ce25c809a3772315d426c3c7a","level":"debug","message":"do request","request.header.accept":"application/vnd.docker.container.image.v1+json, */*","request.header.user-agent":"containerd/1.5.5+unknown","request.method":"GET","severity":"DEBUG","time":"2021-12-21T07:05:35Z","url":"https://eu.gcr.io/v2/gitpod-core-dev/build/image-builder-mk3/bob/blobs/sha256:c6403c89f512020e7e2c34422eb19220f9d61c9ce25c809a3772315d426c3c7a"}
registry-facade-f26zv registry-facade {"digest":"sha256:c6403c89f512020e7e2c34422eb19220f9d61c9ce25c809a3772315d426c3c7a","level":"debug","message":"fetch response received","response.header.accept-ranges":"bytes","response.header.age":"2008","response.header.alt-svc":"h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"","response.header.cache-control":"public, max-age=3600","response.header.content-length":"5698","response.header.content-type":"application/octet-stream","response.header.date":"Tue, 21 Dec 2021 06:32:07 GMT","response.header.etag":"\"9200dc83948060cf66e9f42d3ba012b7\"","response.header.expires":"Tue, 21 Dec 2021 07:32:07 GMT","response.header.last-modified":"Tue, 07 Dec 2021 17:24:05 GMT","response.header.server":"UploadServer","response.header.x-goog-generation":"1638897845200560","response.header.x-goog-hash":"crc32c=nRhgxw==","response.header.x-goog-hash.1":"md5=kgDcg5SAYM9m6fQtO6AStw==","response.header.x-goog-metageneration":"1","response.header.x-goog-storage-class":"STANDARD","response.header.x-goog-stored-content-encoding":"identity","response.header.x-goog-stored-content-length":"5698","response.header.x-guploader-uploadid":"ADPycdsTMR302QU4Ku022VhiIOwRjnZF72Hb6uphGWfjcSPbZt6WKoVATI_vN6BfVtcL9FPjNitiaQ5CO9J8GgOfR5I","response.status":"200 OK","severity":"DEBUG","time":"2021-12-21T07:05:35Z","url":"https://eu.gcr.io/v2/gitpod-core-dev/build/image-builder-mk3/bob/blobs/sha256:c6403c89f512020e7e2c34422eb19220f9d61c9ce25c809a3772315d426c3c7a"}
registry-facade-f26zv registry-facade {"instanceId":"51a12ba3-9d99-4e5a-8ff0-18b7a931b3c6","level":"debug","message":"get manifest (end)","serviceContext":{"service":"registry-facade","version":"commit-075a12d448c9205dd995e1acff4df74f46569b44"},"severity":"DEBUG","spec":{"base_ref":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44","ide_ref":"eu.gcr.io/gitpod-core-dev/build/image-builder-mk3/bob:075a12d448c9205dd995e1acff4df74f46569b44"},"tag":"latest","time":"2021-12-21T07:05:35Z","userId":"","workspaceId":""}
registry-6bd464448c-xv588 docker-registry time="2021-12-21T07:05:36.457667641Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77" err.message="manifest unknown" go.version=go1.11.2 http.request.host=registry.my.domain.com http.request.id=8d132e86-b17b-40a1-9408-041fa33442c9 http.request.method=HEAD http.request.remoteaddr=10.42.2.2 http.request.uri="/v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77" http.request.useragent="containerd/1.5.5+unknown" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.0967ms http.response.status=404 http.response.written=154 vars.name=workspace-images vars.reference=e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77
registry-6bd464448c-xv588 docker-registry 10.42.3.30 - - [21/Dec/2021:07:05:36 +0000] "HEAD /v2/workspace-images/manifests/e764b1a602dc4123d9a027358367b5c4a35b62043e3c630731702144b7d37f77 HTTP/2.0" 404 154 "" "containerd/1.5.5+unknown"
registry-6bd464448c-xv588 docker-registry 10.42.4.1 - - [21/Dec/2021:07:05:38 +0000] "GET / HTTP/2.0" 200 0 "" "kube-probe/1.21"
registry-6bd464448c-xv588 docker-registry 10.42.4.1 - - [21/Dec/2021:07:05:38 +0000] "GET / HTTP/2.0" 200 0 "" "kube-probe/1.21"
Workspace affected
No response
Expected behavior
No response
Example repository
No response
Anything else?
No response
Metadata
Metadata
Assignees
Labels
No labels