Update preview VM image #11053

ArthurSens · 2022-06-30T19:16:11Z

Signed-off-by: ArthurSens [email protected]

Description

Updates the VM image to gitpod-k3s-202206291903

Related Issue(s)

How to test

Release Notes

NONE

Documentation

Werft options:

/werft with-preview

ArthurSens · 2022-06-30T19:16:35Z

/hold

Putting a hold until both teams tested properly that this works 😬

kylos101 · 2022-06-30T19:33:08Z

@ArthurSens Weird, I am not seeing a Results tab, with a link to the preview environment here. 🤔

werft-gitpod-dev-com · 2022-06-30T19:48:43Z

started the job as gitpod-build-as-updae-vm.1 because the annotations in the pull request description changed
(with .werft/ from main)

ArthurSens · 2022-06-30T20:01:23Z

@ArthurSens Weird, I am not seeing a Results tab, with a link to the preview environment here. 🤔

@kylos101 you should get a new one by ticking/unticking the with-preview checkbox in the PR description :)

ArthurSens · 2022-07-01T11:30:21Z

@kylos101 @meysholdt are we confident on merging this one?

Furisto · 2022-07-01T14:39:45Z

@kylos101 @ArthurSens The cgroup filesystem in the preview environment is still v1.

kylos101

@ArthurSens given the feedback from @Furisto , something is not right.

@Furisto how are you checking that the file system is still on cgroup v1?

utam0k · 2022-07-03T23:40:53Z

@Furisto how are you checking that the file system is still on cgroup v1?

@kylos101
If you find /sys/fs/cgroup/cpu path, it means you use cgroup v1.

Furisto · 2022-07-04T08:01:39Z

@Furisto how are you checking that the file system is still on cgroup v1?

Opened a workspace and then did as @utam0k described.

ArthurSens · 2022-07-04T17:44:07Z

@kylos101 @Furisto, I'm not sure how to proceed here.

Do you have a way to certify that a VM image belongs to a certain github release? How do we make sure we have the correct image in place?

And regarding group v2, do you configure something to make it available in production? Am I missing something here?

Furisto · 2022-07-04T22:10:21Z

This is how we activate cgroup v2 in prod: https://github.com/gitpod-io/gitpod-packer-gcp-image/blob/a76de6e8bd479e4f441d5e86affc642abf15c246/setup.sh#L115

kylos101 · 2022-07-06T16:08:55Z

@ArthurSens do you need anything else to test this?

@jenting @utam0k do you have any other recommendations for @ArthurSens ? I ask because I know you wrote this internal doc last night.

ArthurSens · 2022-07-06T18:48:21Z

@ArthurSens do you need anything else to test this?

@jenting @utam0k do you have any other recommendations for @ArthurSens ? I ask because I know you wrote this internal doc last night.

The internal doc written by toru does help a little bit, but the need for a reboot makes things very complicated to add into our CI at the moment... I'll try to take another look at this tomorrow 🤔

jenting · 2022-07-07T02:20:42Z

@ArthurSens do you need anything else to test this?
@jenting @utam0k do you have any other recommendations for @ArthurSens ? I ask because I know you wrote this internal doc last night.

The internal doc written by toru does help a little bit, but the need for a reboot makes things very complicated to add into our CI at the moment... I'll try to take another look at this tomorrow 🤔

Thank you, Arthur. It would be good to support reboot because we are thinking that having a werft annotation to make us able to switch from cgroup v1 or cgroup v2.

utam0k · 2022-07-07T05:35:48Z

@ArthurSens Can we at least decide when we first create the preview-env?

meysholdt · 2022-07-07T12:46:49Z

Thank you, Arthur. It would be good to support reboot because we are thinking that having a werft annotation to make us able to switch from cgroup v1 or cgroup v2.

Does "having a werft annotation to make us able to switch from cgroup v1 or cgroup v2" really require support for reboot? Assumtion: What we want to have here is the ability to pass Linux kernel boot parameters from Werft to the VM as part of the VM-creation-process.

kylos101 · 2022-07-07T17:00:39Z

👋 let's try to keep this like a 🛹

My recommendation is:

The default be set to cgroup v2
If someone wants to test on cgroup v1, they can follow the Notion doc that JenTing and Toru mention here

If following the document to test on cgroup v1 happens often, and it'd save people time in the future, then we can consider the werft annotation in a future PR and issue.

In other words, right now, its too early to say whether we truly need a werft annotation to switch between types of cgroup for a preview environment instance. It'd be better (best?) to save that energy for the Platform Team, so they can work other things.

ArthurSens · 2022-07-07T17:04:54Z

👋 let's try to keep this like a 🛹

My recommendation is:

The default be set to cgroup v2

If someone wants to test on cgroup v1, they can follow the Notion doc that JenTing and Toru mention here

If following the document to test on cgroup v1 happens often, and it'd save people time in the future, then we can consider the werft annotation in a future PR and issue.

In other words, right now, its too early to say whether we truly need a werft annotation to switch between types of cgroup for a preview environment instance. It'd be better (best?) to save that energy for the Platform Team, so they can work other things.

The problem here is that looks like a reboot is needed mid-CI to enable cgroupv2, and that is quite complex to implement.

We depend on the VM being up for several steps of the CI. I'm not saying it is impossible, but will require a fair amount of refactoring 🙃

Is there any other way of enabling chroupv2 that won't require a reboot?

utam0k · 2022-07-07T23:04:14Z

AFAI, there is no way without reboot because cgroup is a core feature of linux kernel. All processes belong to some cgroup. It is probably impossible to switch this without a restart.

Is there any other way of enabling chroupv2 that won't require a reboot?

kylos101 · 2022-07-12T20:41:51Z

@ArthurSens 👋 I'm not sure why the reboot is necessary if you're using our updated image. Is Harvester mutating this grub entry? My understanding based on this comment is that on boot-up, we'd be using cgroup v2.

Can you share the output of cat /etc/default/grub?

meysholdt · 2022-07-14T13:31:56Z

/werft with-preview=true

👎 unknown command: with-preview=true
Use /werft help to list the available commands

meysholdt · 2022-07-14T13:32:12Z

/werft with-preview

👎 unknown command: with-preview
Use /werft help to list the available commands

meysholdt · 2022-07-14T13:32:25Z

/werft run with-preview=true

👍 started the job as gitpod-build-as-updae-vm.8
(with .werft/ from main)

meysholdt · 2022-07-14T16:32:46Z

I got a bit further:
When starting the preview env via werft run github (instead of from this PR) then the preview env gets the new image from this PR.

This can be confirmed by looking at the VM in Harvester and seeing the image version:

in the new VM, the test passes:

ls /sys/fs/cgroup/cgroup.controllers
/sys/fs/cgroup/cgroup.controllers

However, no pods in k3s start.

Some errors from journalctl:

I0714 15:39:44.950125  259700 codec.go:117] "Using lenient decoding as strict decoding failed" err="strict decoding error: unknown field \"conntrack\""
time="2022-07-14T15:39:44Z" level=error msg="Failed to record snapshots for cluster: No snapshot configmap found"
time="2022-07-14T15:39:48Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
E0714 15:39:50.318989  259700 kubelet.go:2391] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

For some reason, calico does not get installed.

If I install it by running kubectl apply -f /var/lib/gitpod/manifests/calico.yaml, it does not launch and kubectl -n kube-system describe pod calico-node-f8gxr shows error:

failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format "slice:prefix:name" for systemd cgroups, got "/kubepods/burstable/pod95dd2279-3853-4cdb-b25a-718c82d4f907/e8f0110ff17cebbe19c298f684bbb2781a308c22b25f3a015a1b9aa9b551c6ee" instead: unknown

Any hints why k3s does not start as expected are very much appreciated :)

kylos101 · 2022-07-14T17:00:56Z

@meysholdt do you have the same trouble when using the latest image? That would be this one.

aledbf · 2022-07-14T18:41:20Z

@meysholdt besides @kylos101 comment, please make sure k3s is started with the flag --kubelet-arg cgroup-driver=systemd

vulkoingim · 2022-07-21T15:50:32Z

.werft/vm/manifests.ts

+      kubectl apply -f /var/lib/gitpod/manifests/csi-driver.yaml
+      kubectl apply -f /var/lib/gitpod/manifests/csi-config.yaml


These paths got changed in https://github.com/gitpod-io/gitpod-packer-gcp-image/pull/114

vulkoingim · 2022-07-21T16:28:24Z

With the latest image and a fresh VM now the preview env succeeds:

https://werft.gitpod-dev.com/job/gitpod-custom-as-updae-vm.11

grep cgroup /proc/filesystems

nodev   cgroup
nodev   cgroup2
...
ls /sys/fs/cgroup/cgroup.controllers
/sys/fs/cgroup/cgroup.controllers

time="2022-07-21T16:02:03Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd ....

meysholdt

Thank you, @vulkoingim!

I can confirm that the preview env on this PR runs on image default/gitpod-k3s-202207120820.qcow2 and that I could successfully start workspace inside that preview env.

jenting · 2022-07-25T09:16:15Z

/werft run with-preview=true

👍 started the job as gitpod-build-as-updae-vm.19
(with .werft/ from main)

vulkoingim · 2022-07-25T09:24:07Z

@jenting this job won't run with the changes, it has to be started manually from the CLI. I'll do it in a second and send a link, after I clean the current VM.

jenting · 2022-07-25T09:28:10Z

@jenting this job won't run with the changes, it has to be started manually from the CLI. I'll do it in a second and send a link, after I clean the current VM.

Thank u

vulkoingim · 2022-07-25T09:29:58Z

https://werft.gitpod-dev.com/job/gitpod-custom-as-updae-vm.12

https://harvester.gitpod-dev.com/dashboard/c/local/harvester/kubevirt.io.virtualmachine/preview-as-updae-vm/as-updae-vm#basics

jenting · 2022-07-25T10:05:04Z

I could open the workspace, and the underlying is cgroupv2 and with cgroup driver systemd.

jenting · 2022-07-25T10:05:11Z

/unhold

kylos101

👍 to unblock this, thanks @Furisto and @jenting !

ArthurSens requested review from a team June 30, 2022 19:16

roboquat added release-note-none size/XS labels Jun 30, 2022

github-actions bot added team: devx team: workspace Issue belongs to the Workspace team labels Jun 30, 2022

roboquat added the do-not-merge/hold label Jun 30, 2022

jenting approved these changes Jul 1, 2022

View reviewed changes

vulkoingim approved these changes Jul 1, 2022

View reviewed changes

kylos101 requested changes Jul 1, 2022

View reviewed changes

ArthurSens force-pushed the as/updae-vm branch from 4e4069b to 72d8994 Compare July 6, 2022 17:57

meysholdt force-pushed the as/updae-vm branch from 72d8994 to bc3fd9c Compare July 14, 2022 14:55

roboquat added size/S and removed size/XS labels Jul 21, 2022

Update preview env image, enabling cgroupv2

40ec2b2

vulkoingim force-pushed the as/updae-vm branch from 580225b to 40ec2b2 Compare July 21, 2022 15:49

vulkoingim reviewed Jul 21, 2022

View reviewed changes

vulkoingim requested review from kylos101 and a team July 21, 2022 16:28

meysholdt approved these changes Jul 22, 2022

View reviewed changes

roboquat removed the do-not-merge/hold label Jul 25, 2022

Furisto approved these changes Jul 25, 2022

View reviewed changes

kylos101 approved these changes Jul 25, 2022

View reviewed changes

roboquat merged commit 70d0392 into main Jul 25, 2022

roboquat deleted the as/updae-vm branch July 25, 2022 14:08

roboquat added the deployed: workspace Workspace team change is running in production label Aug 3, 2022

		kubectl apply -f /var/lib/gitpod/manifests/csi-driver.yaml
		kubectl apply -f /var/lib/gitpod/manifests/csi-config.yaml

Update preview VM image #11053

Update preview VM image #11053

Uh oh!

Conversation

ArthurSens commented Jun 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

How to test

Release Notes

Documentation

Werft options:

Uh oh!

ArthurSens commented Jun 30, 2022 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kylos101 commented Jun 30, 2022

Uh oh!

werft-gitpod-dev-com bot commented Jun 30, 2022

Uh oh!

ArthurSens commented Jun 30, 2022 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ArthurSens commented Jul 1, 2022

Uh oh!

Furisto commented Jul 1, 2022

Uh oh!

kylos101 left a comment

Choose a reason for hiding this comment

Uh oh!

utam0k commented Jul 3, 2022

Uh oh!

Furisto commented Jul 4, 2022

Uh oh!

ArthurSens commented Jul 4, 2022

Uh oh!

Furisto commented Jul 4, 2022

Uh oh!

kylos101 commented Jul 6, 2022

Uh oh!

ArthurSens commented Jul 6, 2022 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jenting commented Jul 7, 2022 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

utam0k commented Jul 7, 2022

Uh oh!

meysholdt commented Jul 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kylos101 commented Jul 7, 2022

Uh oh!

ArthurSens commented Jul 7, 2022 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

utam0k commented Jul 7, 2022

Uh oh!

kylos101 commented Jul 12, 2022

Uh oh!

meysholdt commented Jul 14, 2022 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

meysholdt commented Jul 14, 2022 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

meysholdt commented Jul 14, 2022 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

meysholdt commented Jul 14, 2022

Uh oh!

kylos101 commented Jul 14, 2022

Uh oh!

aledbf commented Jul 14, 2022

Uh oh!

vulkoingim Jul 21, 2022

Choose a reason for hiding this comment

Uh oh!

vulkoingim commented Jul 21, 2022

Uh oh!

ArthurSens commented Jun 30, 2022 •

edited

Loading

ArthurSens commented Jun 30, 2022 •

edited by werft-gitpod-dev-com bot

Loading

ArthurSens commented Jun 30, 2022 •

edited by werft-gitpod-dev-com bot

Loading

ArthurSens commented Jul 6, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jenting commented Jul 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading

meysholdt commented Jul 7, 2022 •

edited

Loading

ArthurSens commented Jul 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading

meysholdt commented Jul 14, 2022 •

edited by werft-gitpod-dev-com bot

Loading

meysholdt commented Jul 14, 2022 •

edited by werft-gitpod-dev-com bot

Loading

meysholdt commented Jul 14, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jenting commented Jul 25, 2022 •

edited by werft-gitpod-dev-com bot

Loading