dynamic overlays

[cgwalters]

This overlaps with #7 some.

Here, the basic idea is something like:

bootc config add [--root=/etc] https://examplecorp.com/config.yml

(OR with support for OCI Artifacts we support bootc config add [--root=/etc] registry:quay.io/examplecorp/config-server-base:latest)

Where config.yml is a standard Kubernetes ConfigMap. By default, we "mount" the keys to /etc. Then, bootc upgrade looks for updates to all provided configmaps - if any change, it triggers the same upgrade logic as the base image.

We also fetch and handle secret objects in the same way. It'd be cool though to support something like handling encrypted secrets (and configmaps) which need to be decrypted via a key (which could be in a TPM or so).

We also need to think carefully about file permissions; mode 0644 for all configmap files and 0600 for secrets may make sense. In addition we could support special annotations to override these.

(This should also work to be invoked immediately after bootc install to have it ready on the first boot, i.e. we also have a --root argument or so)

[cgwalters]

To flesh this out a bit more, bootc upgrade would check for updates to these things too.

Also, another variant of this is that we should support fetching all this stuff from the cloud user data too.

[cgwalters]

So this design would work in an obvious way outside of Kubernetes - we're using a standard API type, but we're not depending on an API server or kubelet, etc.

But, in a Kubernetes context, it would make total sense to fetch this data from the API server instead of just running http. (And potentially use proper watches to react to events instead of polling). I'm thinking we probably want to make the mechanism to fetch this data pluggable; ultimately it might be that there's /var/lib/bootc/config.d and a privileged container image could e.g. use kubelet's credentials to fetch from the API server instead, and just hand things down to bootc.

This intersects with openshift/machine-config-operator#3327 too.

[cgwalters]

This intersects with openshift/machine-config-operator#3327 too.

A hugely interesting topic here of course is whether we should have a declarative way to set this stuff up. Today for example we have an imperative interface in bootc switch to specify the root image.

But...following the kubelet "static pod" model, and building on the pod analogy above, perhaps we have
/etc/bootc/host.yaml which is a subset of the pod spec, and allows declaratively writing the attached configmaps.

(This topic also relates to coreos/rpm-ostree#2326 )

[cgwalters]

It may also be interesting here to try to support something like an annotation which says the configmap changes should be live-applied by default - in the general case we'd probably need some sort of "post-apply hook" to ask affected services to reload. Or, such services could use inotify of course (at the risk of seeing partial states).

[cgwalters]

To implement this I'm thinking:

• Add a kubernetes feature (on by default)
• Use the https://docs.rs/k8s-openapi/latest/k8s_openapi/ crate
• Support basic fetch via http, and investigate OCI artifact support (going to need to be driven into skopeo most likely)
• Write configmaps as ostree commits (map name to ref, etc.); although we may want to actually split into "metadata" and "contents"; metadata changes to the configmap shouldn't appear as changes probably, except things like the potential update semantics
• Create merged commit by applying configmap commits into /usr/etc to start - and here we have the interesting question of handling conflicts (i.e. what happens if my configmap has /etc/sudoers and the host has that file too? I'd probably vote that we error by default and require an annotation to overwrite?)
• bootc update looks for changes to base and all configmaps

Now...what many people will want is a read-only /etc; if we have that, it becomes logically easier to have each configmap be a mounted tmpfs or whatever, which we can more easily implement individual live-apply semantics.

[cgwalters]

Trying to play around with this, the ergonomics are definitely somewhat annoying because configmaps can't have / in keys. If one wanted to e.g. write drop a a CA certificate into /etc/pki/ca-trust/source/anchors and also a local registry mirror config in /etc/containers/registries.conf.d it'd require separate configmaps with separate mountpoints. Maybe we just accept that for now.

Edit: Or, we could accept a List object of configmaps.