genpolicy: add utility script for containerd pull #213

Redent0r · 2024-08-05T17:40:51Z

Add script that helps adapt containerd and docker config for containerd pull feature to work as expected.

Merge Checklist

Followed patch format from upstream recommendation: https://github.com/kata-containers/community/blob/main/CONTRIBUTING.md#patch-format
- Included a single commit in a given PR - at least unless there are related commits and each makes sense as a change on its own.
Aware about the PR to be merged using "create a merge commit" rather than "squash and merge" (or similar)
genPolicy only: Ensured the tool still builds on Windows
The upstream/missing label (or upstream/not-needed) has been set on the PR.

Summary

Add script that helps adapt containerd and docker config for containerd pull feature to work as expected.

Test Methodology

Tested locally that I'm able to authenticate to private registry after running the script

Useful for debugging. Signed-off-by: Dan Mihai <[email protected]>

Make sure the hash of an incoming Policy matches the value of the SNP Host Data field. The value of Host Data will be validated through Remote Attestation, outside of this patch. Signed-off-by: Dan Mihai <[email protected]>

There are 10 segments in the ACPI tables, and CLH works better when it uses all of them. Signed-off-by: Dan Mihai <[email protected]>

When a request cannot be evaluated to true, OPA can return an empty response. It doesn't respond with "response = false" unless a default value of false has been defined. Handle empry responses the same way as "response = false", thus allowing users to bypass those responses by using AllowRequestsFailingPolicy := true. Signed-off-by: Dan Mihai <[email protected]>

This is needed when enabling dm-verity. `udevd` reads kernel uevents that announce the creation of `/dev/dm-XXX` devices, and then creates devices with the actual names under `/dev/mapper/`. Signed-off-by: Wedson Almeida Filho <[email protected]>

This allows us to avoid repeating paths when they're the same. Signed-off-by: Wedson Almeida Filho <[email protected]>

This replicates Wedson's changes in 0935263 in a way that is aligned with the upstream implementation introduced in kata-containers#7200. NOTE: This will require compiling the runtime with DEFSHAREDFS_CLH_SNP_VIRTIOFS=none.

Upstream now uses the new DEFSTATICRESOURCEMGMT_TEE variable to set static resource management for TEEs so we align on that. It's true by default so we don't have to update our build script for this. NOTE: For non-tee CH, upstream now uses DEFSTATICRESOURCEMGMT_CLH (already in our codebase) instead of DEFSTATICRESOURCEMGMT. It's still false by default so we WILL have to update our build script for this one.

The layer string is now base64-encoded, so decode it before inspecting the fields. Signed-off-by: Wedson Almeida Filho <[email protected]>

Newer versions of depmod are failing without the -a option. They get confused with the kernel version and expect it to start with a slash: depmod: FATAL: modules: not absolute path. Signed-off-by: Wedson Almeida Filho <[email protected]>

This allows the agent to ensure the integrity of the device. Signed-off-by: Wedson Almeida Filho <[email protected]>

This allows snapshotters to skip the path to layers. They can, naturally, still specify the full path to other locations when needed. Signed-off-by: Wedson Almeida Filho <[email protected]>

This allows us to tell, from the error, what the agent was attempting to do with the devicemapper. Signed-off-by: Wedson Almeida Filho <[email protected]>

This is so that dependents are removed first, so that depencies don't have references anymore when they're unmounted. Signed-off-by: Wedson Almeida Filho <[email protected]>

- Add missing systemd command line parameters for initrd images

An upstream change in CBL-Mariner now requires the UVM to have the zstd-libs package as a dependency of systemd

Pick up genpolicy improvements from the main branch. Signed-off-by: Dan Mihai <[email protected]>

Allow genpolicy to process Pod YAML files including topologySpreadConstraints. Signed-off-by: Dan Mihai <[email protected]>

Add Policy doc. Signed-off-by: Dan Mihai <[email protected]>

This partially applies 2b5c84b. Adding SNP template for config, and values to the runtime makefile to generate the output toml. Based on [Joana's SNP Guide](https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-run-kata-containers-with-SNP-VMs.md). Signed-Off-By: Alex Carter <[email protected]> Signed-Off-By: Aurelien Bombo <[email protected]>

This is so that file systems don't fail when we pass kata-specific options from the snapshotter to kata. We already had this upstreamed in kata-containers@6163c35, however upstream broke us in kata-containers@8392c71: `parse_mount_flags_and_options()` was replaced with a new function `parse_mount_options()` which didn't take `io.katacontainers.*` options into account.

* WIP almost compiles * Add special fields * remove dependency on /protocol * Remove special fields * renamy my_agent to agent * clean up * update busybox sample * remove unused struct

containerd needs this in some cases when using volume mounts, so implement it to unblock containerd. Signed-off-by: Wedson Almeida Filho <[email protected]>

genpolicy: add persistent storage support for stateful sets

Add common images category Signed-off-by: Saul Paredes <[email protected]>

The two samples define Pod overhead values. When testing these samples on clusters where different podOverhead values are defined by the respective runtime classes, the tests fail as the values need to match, see: /plugin/pkg/admission/runtimeclass/admission.go in the kubernetes GitHub repository. Hence, removing the overhead definitions. Re-ran update_policy_samples.py. No change. Signed-off-by: Manuel Huber <[email protected]>

samples: add common images category

genpolicy: update policy samples

For docker-based builds only install Rust when necessary. Further, execute the detect Rust version check only when intending to install Rust. As of today, this is the case when we intend to build the agent during rootfs build. Signed-off-by: Manuel Huber <[email protected]>

We set the VERSION variable consistently across Makefiles to 'unknown' if the file is empty or not present. We also use git commands consistently for calculating the COMMIT, COMMIT_NO variables, not erroring out when building outside of a git repository. In create_summary_file we also account for a missing/empty VERSION file. This makes e.g. the UVM build process in an environment where we build outside of git with a minimal/reduced set of files smoother. Signed-off-by: Manuel Huber <[email protected]>

Avoid hitting docker.io by using mcr.microsoft.com/acc/samples/acc-perl:1.0. Signed-off-by: Dan Mihai <[email protected]>

samples: change job.yaml to an mcr container image

Add the new category for tests we want to run genpolicy against, but not run them in deployments (as the name suggests, we cannot run those). Signed-off-by: Manuel Huber <[email protected]>

samples: introduce incomplete_init category

Cherry-pick upstream PR kata-containers#9825: osbuilder: allow rootfs builds w/o git or version file deps

- Support for Mariner 3 builds using OS_VERSION variable - Improvements to IGVM build process and flow as described in README - Adoption of using only cloud-hypervisor-cvm on CBL-Mariner Signed-off-by: Manuel Huber <[email protected]>

tools: Improve igvm-builder and node-builder/azure-linux scripting

At the moment, we have circular dependencies between tardev-snapshotter.service and containerd.service. Specifically, containerd.service needs tardev-snapshotter.service to run any CC pods, while tardev-snapshotter.service needs containerd.service to download image layers. This dependency will be eliminated once we switch to using remote-snapshotter. Currently, tardev-snapshotter.service's binding to containerd.service gets delayed, and we won't be able to run any CC pods until the boot process is completed. It doesn't matter which service starts first. Based on the current logic, it makes more sense to use WantedBy=kubelet.service in tardev-snapshotter.service, as we won't be able to start any CC pods without kubelet. In the future, once tardev-snapshotter becomes a remote snapshotter again, it will make more sense to use WantedBy=containerd.service. Signed-off-by: Mitch Zhu <[email protected]>

tardev: update tardev-snapshotter.service

Use container image sources from ACR/MCR. Signed-off-by: Manuel Huber <[email protected]>

samples: reduce dependencies to docker hub

Add script that helps adapt containerd and docker config for containerd pull feature to work as expected. Signed-off-by: Saul Paredes <[email protected]>

src/tools/genpolicy/setup_containerd_docker.sh

danmihai1 · 2024-08-05T23:11:00Z

src/tools/genpolicy/genpolicy-advanced-command-line-parameters.md

-You may specify `-d` to use existing `containerd` installation as image manager. This method supports a wider set of images (e.g., older images with `v1` manifest). Needs `sudo` permission to access socket - e.g.,
+# Use containerd to pull and manage images (required for managed identity based authentication)
+
+Prereq: This features needs to run the following script to adapt your docker and containerd config (needs `sudo` access):


I think these changes are too intrusive to push average users to use this script.

For average users, I would prefer a list of steps in an MD file - similar to the steps from https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/containerd-kata.md. Also, I would like to update such doc Upstream instead of pushing just MSFT users in this direction.

But let's talk with other folks in our team too - maybe they have different opinions.

I agree it's too much to require them to use this. This might exist for users that don't feel like doing all these steps, but they are also welcome to do them on their own. I'll add better instructions to the MD, and open a PR upstream with all squashed changes

I disagree with:

Calling this script a prerequisite. We should say something like: "if you need help with your containerd settings, see this doc for related tips".

Hiding a bunch of details in a script. Users that want to perform these steps should review the doc and perform just those steps that look reasonable to them.

Signed-off-by: Saul Paredes <[email protected]>

Add guide on how to setup containerd and docker for genpolicy containerd pull feature. Signed-off-by: Saul Paredes <[email protected]>

danmihai1 · 2024-08-07T21:12:10Z

src/tools/genpolicy/genpolicy-advanced-command-line-parameters.md

-You may specify `-d` to use existing `containerd` installation as image manager. This method supports a wider set of images (e.g., older images with `v1` manifest). Needs `sudo` permission to access socket - e.g.,
+# Use containerd to pull and manage images (required for managed identity based authentication)
+
+Prereq: This features needs to run the following script to adapt your docker and containerd config (needs `sudo` access):


I disagree with:

Calling this script a prerequisite. We should say something like: "if you need help with your containerd settings, see this doc for related tips".

Hiding a bunch of details in a script. Users that want to perform these steps should review the doc and perform just those steps that look reasonable to them.

danmihai1 · 2024-08-07T21:16:57Z

docs/how-to/setup-containerd-pull.md

@@ -0,0 +1,96 @@
+# Manual Steps for Setting Up Containerd and Docker for Genpolicy Tool


The name of this doc should include "genpolicy".

danmihai1 · 2024-08-07T21:18:35Z

docs/how-to/setup-containerd-pull.md

+sudo apt-get install -y containerd
+```
+
+#### Using `tdnf` (for Photon OS-based systems)


This is not an automated script, so it's OK to keep just the Ubuntu example. That's one advantage of providing a doc instead a script.

danmihai1 · 2024-08-07T21:19:20Z

docs/how-to/setup-containerd-pull.md

+## Steps
+
+### 1. Install Containerd
+


Add the OS version that you used for testing - because other versions might not work, and that's OK.

danmihai1 · 2024-08-07T21:20:49Z

docs/how-to/setup-containerd-pull.md

+
+The socket file is usually `/run/containerd/containerd.sock` or `/var/run/containerd/containerd.sock`
+
+### 6. Fix Containerd Socket File Permissions


This is not a "fix" - it's just something that might be useful for genpolicy users.

danmihai1 and others added 30 commits December 19, 2023 13:41

rootfs: add back coreutils to mariner's image (#48)

661c00f

Useful for debugging. Signed-off-by: Dan Mihai <[email protected]>

rootfs: add back ps to mariner's rootfs (#49)

232714c

Useful for debugging. Signed-off-by: Dan Mihai <[email protected]>

runtime: agent: SNP HOST_DATA set-up (#43)

5fe97f7

runtime: agent: validate policy contents (#50)

e25109d

Make sure the hash of an incoming Policy matches the value of the SNP Host Data field. The value of Host Data will be validated through Remote Attestation, outside of this patch. Signed-off-by: Dan Mihai <[email protected]>

runtime: use 10 PCI segments for the Guest (#59)

8d13eb1

There are 10 segments in the ACPI tables, and CLH works better when it uses all of them. Signed-off-by: Dan Mihai <[email protected]>

runtime: add support for layer-src-prefix option

b60201a

This allows us to avoid repeating paths when they're the same. Signed-off-by: Wedson Almeida Filho <[email protected]>

runtime: Allow disabling FS sharing for CLH/SNP

520e73b

This replicates Wedson's changes in 0935263 in a way that is aligned with the upstream implementation introduced in kata-containers#7200. NOTE: This will require compiling the runtime with DEFSHAREDFS_CLH_SNP_VIRTIOFS=none.

runtime: decode layer string

e04ae70

The layer string is now base64-encoded, so decode it before inspecting the fields. Signed-off-by: Wedson Almeida Filho <[email protected]>

osbuilder: include tardev components to rootfs

eb3f993

Newer versions of depmod are failing without the -a option. They get confused with the kernel version and expect it to start with a slash: depmod: FATAL: modules: not absolute path. Signed-off-by: Wedson Almeida Filho <[email protected]>

agent: use dm-verity if io.katacontainers.fs-opt.root-hash is set

9f386d9

This allows the agent to ensure the integrity of the device. Signed-off-by: Wedson Almeida Filho <[email protected]>

agent: change to layer directory when mounting overlays

1b01983

This allows snapshotters to skip the path to layers. They can, naturally, still specify the full path to other locations when needed. Signed-off-by: Wedson Almeida Filho <[email protected]>

agent: add some context to device mapper error

7542d84

This allows us to tell, from the error, what the agent was attempting to do with the devicemapper. Signed-off-by: Wedson Almeida Filho <[email protected]>

agent: unmount storages in reverse order

5338761

This is so that dependents are removed first, so that depencies don't have references anymore when they're unmounted. Signed-off-by: Wedson Almeida Filho <[email protected]>

runtime: minimum 10 seconds timeout for bootvm (#37)

c70d40a

runtime: accomodate slower than expected boot (#44)

2eda1f1

Append systemd kernel cmdline params for initrd (#39)

93c64cd

- Add missing systemd command line parameters for initrd images

tools: keep package zstd-libs in the UVM (#104)

19aa313

An upstream change in CBL-Mariner now requires the UVM to have the zstd-libs package as a dependency of systemd

runtime: enable confidential guest by default in non-SNP config (#112)

ae75599

tools: pick up genpolicy improvements (#114)

820aaac

Pick up genpolicy improvements from the main branch. Signed-off-by: Dan Mihai <[email protected]>

genpolicy: add topologySpreadConstraints support (#115)

29f4aef

Allow genpolicy to process Pod YAML files including topologySpreadConstraints. Signed-off-by: Dan Mihai <[email protected]>

docs: add agent policy how-to doc (#116)

e153ece

Add Policy doc. Signed-off-by: Dan Mihai <[email protected]>

Add snapshotter build files to .gitignore

39041bc

Add src/agent/samples/policy/test-input/ to .gitignore

3286dc6

genpolicy: Readd agent type definitions (#118)

bdd4585

* WIP almost compiles * Add special fields * remove dependency on /protocol * Remove special fields * renamy my_agent to agent * clean up * update busybox sample * remove unused struct

utarfs: implement the enumeration of xattrs (#119)

3683d4c

containerd needs this in some cases when using volume mounts, so implement it to unblock containerd. Signed-off-by: Wedson Almeida Filho <[email protected]>

arc9693 and others added 18 commits June 7, 2024 02:31

Merge pull request #199 from microsoft/archana1/add-pv-ss

2d32df1

genpolicy: add persistent storage support for stateful sets

samples: add common images category

980fb87

Add common images category Signed-off-by: Saul Paredes <[email protected]>

Merge pull request #200 from microsoft/saulparedes/test_common_images

12d4585

samples: add common images category

Merge pull request #201 from microsoft/mahuber/samples

a4b4ca2

genpolicy: update policy samples

samples: change job.yaml to an mcr container image

f2f5d12

Avoid hitting docker.io by using mcr.microsoft.com/acc/samples/acc-perl:1.0. Signed-off-by: Dan Mihai <[email protected]>

Merge pull request #207 from microsoft/danmihai1/job-mcr

cd7c5fa

samples: change job.yaml to an mcr container image

samples: introduce incomplete_init category

1590a72

Add the new category for tests we want to run genpolicy against, but not run them in deployments (as the name suggests, we cannot run those). Signed-off-by: Manuel Huber <[email protected]>

Merge pull request #208 from microsoft/mahuber/incomplete_init

cc23bdb

samples: introduce incomplete_init category

Merge pull request #206 from microsoft/mahuber/cherry-pick-upstr

e7cb0d1

Cherry-pick upstream PR kata-containers#9825: osbuilder: allow rootfs builds w/o git or version file deps

Merge pull request #204 from microsoft/mahuber/uvm-spec-build-pr

dd57910

tools: Improve igvm-builder and node-builder/azure-linux scripting

Merge pull request #209 from microsoft/mitchzhu/tardev-service

1276d40

tardev: update tardev-snapshotter.service

samples: reduce dependencies to docker hub

16523e2

Use container image sources from ACR/MCR. Signed-off-by: Manuel Huber <[email protected]>

Merge pull request #210 from microsoft/tests/img-pull-src

386cab0

samples: reduce dependencies to docker hub

Redent0r added the upstream/missing PRs that are yet to be upstreamed label Aug 5, 2024

genpolicy: add utility script for containerd pull

7f872b1

Add script that helps adapt containerd and docker config for containerd pull feature to work as expected. Signed-off-by: Saul Paredes <[email protected]>

Redent0r force-pushed the saulparedes/add_util_setup_script branch from c36d773 to 7f872b1 Compare August 5, 2024 18:44

danmihai1 requested changes Aug 5, 2024

View reviewed changes

Redent0r added 2 commits August 6, 2024 15:38

genpolicy: improve setup script

3e5b2b6

Signed-off-by: Saul Paredes <[email protected]>

genpolicy: add guide for setup containerd pull

0b22797

Add guide on how to setup containerd and docker for genpolicy containerd pull feature. Signed-off-by: Saul Paredes <[email protected]>

danmihai1 requested changes Aug 7, 2024

View reviewed changes

danmihai1 force-pushed the msft-main branch from d5ec820 to b900faf Compare April 3, 2025 21:04

Redent0r force-pushed the msft-main branch from 7954edb to 9746966 Compare June 5, 2025 17:21

Redent0r force-pushed the msft-main branch from f4b3c21 to 57499f2 Compare June 17, 2025 17:08

manuelh-dev force-pushed the msft-main branch from 57499f2 to c04bfdc Compare June 25, 2025 22:33

manuelh-dev force-pushed the msft-main branch from c04bfdc to 5586d27 Compare July 25, 2025 16:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

genpolicy: add utility script for containerd pull #213

genpolicy: add utility script for containerd pull #213

Uh oh!

Redent0r commented Aug 5, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danmihai1 Aug 5, 2024

Uh oh!

Redent0r Aug 6, 2024

Uh oh!

danmihai1 Aug 7, 2024

Uh oh!

danmihai1 Aug 7, 2024

Uh oh!

danmihai1 Aug 7, 2024

Uh oh!

danmihai1 Aug 7, 2024

Uh oh!

danmihai1 Aug 7, 2024

Uh oh!

danmihai1 Aug 7, 2024

Uh oh!

Uh oh!

		@@ -0,0 +1,96 @@
		# Manual Steps for Setting Up Containerd and Docker for Genpolicy Tool


		The socket file is usually `/run/containerd/containerd.sock` or `/var/run/containerd/containerd.sock`

		### 6. Fix Containerd Socket File Permissions

genpolicy: add utility script for containerd pull #213

Are you sure you want to change the base?

genpolicy: add utility script for containerd pull #213

Uh oh!

Conversation

Redent0r commented Aug 5, 2024

Merge Checklist

Summary

Test Methodology

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!