dynamic overlays

[cgwalters]

This overlaps with #7 some.

Here, the basic idea is something like:

bootc config add [--root=/etc] https://examplecorp.com/config.yml

(OR with support for OCI Artifacts we support bootc config add [--root=/etc] registry:quay.io/examplecorp/config-server-base:latest)

Where config.yml is a standard Kubernetes ConfigMap. By default, we "mount" the keys to /etc. Then, bootc upgrade looks for updates to all provided configmaps - if any change, it triggers the same upgrade logic as the base image.

We also fetch and handle secret objects in the same way. It'd be cool though to support something like handling encrypted secrets (and configmaps) which need to be decrypted via a key (which could be in a TPM or so).

We also need to think carefully about file permissions; mode 0644 for all configmap files and 0600 for secrets may make sense. In addition we could support special annotations to override these.

(This should also work to be invoked immediately after bootc install to have it ready on the first boot, i.e. we also have a --root argument or so)

[cgwalters]

To flesh this out a bit more, bootc upgrade would check for updates to these things too.

Also, another variant of this is that we should support fetching all this stuff from the cloud user data too.

[cgwalters]

So this design would work in an obvious way outside of Kubernetes - we're using a standard API type, but we're not depending on an API server or kubelet, etc.

But, in a Kubernetes context, it would make total sense to fetch this data from the API server instead of just running http. (And potentially use proper watches to react to events instead of polling). I'm thinking we probably want to make the mechanism to fetch this data pluggable; ultimately it might be that there's /var/lib/bootc/config.d and a privileged container image could e.g. use kubelet's credentials to fetch from the API server instead, and just hand things down to bootc.

This intersects with openshift/machine-config-operator#3327 too.

[cgwalters]

This intersects with openshift/machine-config-operator#3327 too.

A hugely interesting topic here of course is whether we should have a declarative way to set this stuff up. Today for example we have an imperative interface in bootc switch to specify the root image.

But...following the kubelet "static pod" model, and building on the pod analogy above, perhaps we have
/etc/bootc/host.yaml which is a subset of the pod spec, and allows declaratively writing the attached configmaps.

(This topic also relates to coreos/rpm-ostree#2326 )

[cgwalters]

It may also be interesting here to try to support something like an annotation which says the configmap changes should be live-applied by default - in the general case we'd probably need some sort of "post-apply hook" to ask affected services to reload. Or, such services could use inotify of course (at the risk of seeing partial states).

[cgwalters]

To implement this I'm thinking:

• Add a kubernetes feature (on by default)
• Use the https://docs.rs/k8s-openapi/latest/k8s_openapi/ crate
• Support basic fetch via http, and investigate OCI artifact support (going to need to be driven into skopeo most likely)
• Write configmaps as ostree commits (map name to ref, etc.); although we may want to actually split into "metadata" and "contents"; metadata changes to the configmap shouldn't appear as changes probably, except things like the potential update semantics
• Create merged commit by applying configmap commits into /usr/etc to start - and here we have the interesting question of handling conflicts (i.e. what happens if my configmap has /etc/sudoers and the host has that file too? I'd probably vote that we error by default and require an annotation to overwrite?)
• bootc update looks for changes to base and all configmaps

Now...what many people will want is a read-only /etc; if we have that, it becomes logically easier to have each configmap be a mounted tmpfs or whatever, which we can more easily implement individual live-apply semantics.

[cgwalters]

Trying to play around with this, the ergonomics are definitely somewhat annoying because configmaps can't have / in keys. If one wanted to e.g. write drop a a CA certificate into /etc/pki/ca-trust/source/anchors and also a local registry mirror config in /etc/containers/registries.conf.d it'd require separate configmaps with separate mountpoints. Maybe we just accept that for now.

Edit: Or, we could accept a List object of configmaps.

[cgwalters]

Also, we should add support for kernel arguments via ConfigMap too...it'd be a really good match.

[cgwalters]

Dug into this a bit more, and actually we have all the bits we need for client-side support for configmaps-in-registry today, which is awesome.

The main gap is documenting support for uploading them (xref containers/buildah#5091 )

[fabiendupont]

@cgwalters, apart from being closer to existing Kubernetes objects, what ConfigMap/Secret bring compared to systemd-confext?

Systemd expects to find the config extensions in a specific folder, e.g. /run/confexts/. It is possible to ship config OCI images in the OS image and mount them under /run/confexts/ with ComposeFS via systemd.mount units before the systemd-confext.service unit.

[cgwalters]

This is a complex topic. First, I think it makes sense to enable confext use on top of bootc.

However the other case here is I think many will want support for "lifecycle binding" configs and the OS, i.e. bootc update intends to update both as a transactional unit (and integrated with rollbacks by default); confext is an independent thing today from OS updates by default (which is good, we also want something like that).

And yes, there is the Kubernetes orientation versus the DDI stuff.

[cgwalters]

(From an organizational perspective note bootc is part of the containers/ GH org, not coreos/)

bootc.coreos.io/baseDir

Right, this one already exists in the WIP code...I think maybe what we should do is make a shared hackmd doc and refine it into a spec, then update the first comment here or so?

As far as the other metadata...maybe. So far, there hasn't been a use case for it in Kubernetes, right? I think specifically having executables in configmaps is probably something we should think of as an anti-pattern - we have container images for that. This reduces the need significantly for fileMode. The use cases for user/group owners also seem like they need very specific motivation.

[arewm]

To flesh this out a bit more, bootc upgrade would check for updates to these things too.

Would it make sense to enable only the configmap to be updated without updating the bootc image as well or would these actions/lifecycles be bound together somehow?

Would you be able to use an image manifest to store the config maps with the config descriptor could be used to describe how all of the layers should be applied to the file system?

These could then be customized per-architecture and distributed as an image index so a configuration file could be applied to the bootc installations regardless of the architecture.

[jlebon]

Here are some of the ways we've discussed so far in how configs could be packaged:

• configmap (YAML hosted somewhere or in an OCI artifact)
• FROM scratch container image
• systemd-confext (I guess... as an image file hosted somewhere or in an OCI artifact)

ISTM like those are completely orthogonal to how they're bound to the host (in the case of systemd-confext, it could be none at all, but... it could also be "managed" like the other options would be). Even in the managed option, there's e.g. configured in /etc (local state) or in /usr (closer to LBIs).

[jlebon]

I've mentioned this before OOB, but just to surface it here, from the CoreOS side we're very interested in this capability because there the default entrypoint is not to do a custom build, but the day-2 config management is still extremely useful. IOW, we want this so as to reduce the need for either having to reprovision (current Ignition model) or having to maintain a CI pipeline (bootc model).

[jlebon]

We had some in-person discussions related to this last week. Just wanted to jot down some of what was mentioned:

• Strongly related to dynamic overlays #22 (comment) and as mentioned in on-cluster builds enhancement openshift/enhancements#1515, this has important implications for on-cluster layering in OpenShift. Currently we're building a derived images containing MachineConfig content. This means that as soon as a user changes a MachineConfig, we have to trigger a new build. If we instead have a separate way to apply config changes, it means a lot less churn in how often we need to rebuild the full image.
• Following on the previous point, this might be out of scope for this issue but it'd be really nice if this just extended to /var as well. This would allow the MCO to get out of the business of writing files directly almost entirely without breaking backwards compatibility with what is currently possible with MachineConfigs. Obviously the implementation details would be different from the /etc case. E.g. we'd need to also think about reconciliation there, etc...

[jlebon]

Can we rename this issue to e.g.:

Config management decoupled from OS updates

I think the configmap approach is just one potential implementation of this, but we should discuss which implementation is the most appropriate.

[cgwalters]

I think I'd prefer to keep this issue focused on a bootc-oriented deliverable for this.

But it absolutely makes sense to consider things that aren't configmaps or that wouldn't require changes to bootc...we could have a tracker issue somewhere for that? Could be a discussion? On this topic we do have https://docs.fedoraproject.org/en-US/bootc/dynamic-reconfiguration/ today and wherever we take that issue it'd make sense to add that content there (e.g. specifically the advantages/disadvantages of ansible, a custom agent writing files directly, maybe also systemd-confext)

[jlebon]

I think I'd prefer to keep this issue focused on a bootc-oriented deliverable for this.

Agree with this, though that doesn't contradict

I think the configmap approach is just one potential implementation of this, but we should discuss which implementation is the most appropriate.

right?

[cgwalters]

Hmm, perhaps? I mean it kind of depends on the scope of solutions. I initially took your suggestion as being very broad potentially (again things like ansible or whatever).

But do you have an outline of something concrete that might be within bootc's scope that isn't configmap-like?

[jlebon]

Whoops, just realized I never got back to you here.

Briefly, I think configmaps are not rich enough for the purpose of configuring a whole OS. #22 (comment) captures it well I think (and while I agree e.g. executables or non-root uid/gids are not prevalent, they're certainly not uncommon). My understanding also from the k8s side is that configmaps tend to be abused for things that would be better represented as a proper CRD where you can have proper validation and types, etc...

As a first approach, I like better @jmarrero's idea of just shoving configs in a container image. Yes, it's awkward, but also builds easily, composes nicely (you can FROM a config image and add more things), etc... All the things around ownership, file modes, paths, etc... are all first class in tar. And it's trivial enough really that you can assemble it directly rather than actually doing a container build. (And actually... that's something I'm thinking could make sense for the MCO to do.)