install to-disk with LUKS + TPM broken

[jmpolom]

Does bootc install to-disk --block-setup tpm2-luks /dev/diskX actually work? I tried this in a qemu virtual machine with emulated TPM (via swtpm) and while it installed successfully, upon rebooting the VM into the freshly installed OS the systemd-cryptsetup units failed to decrypt the LUKS volume. Has this actually been tested or otherwise known to work? I will try on real hardware but this has me concerned this feature is not really in a functional state.

I tested with vanilla Fedora 39 Server to try and rule out this being related to the use of a virtual machine with emulated TPM. After installing tpm2-tools, adding the tpm2-tss dracut modules, and running systemd-cryptenroll for the LUKS volume I had an installation that repeatedly would unlock automatically via the TPM at boot (no password and no failures). Also tried with Fedora 39 Silverblue (added modules to initrd and enabled custom initramfs generation with rpm-ostree) -- same results. In both cases the LUKS volume was enrolled after the installed OS was provisioned and booted for the first time although I really doubt that has any effect on anything. I do not believe the test setup (IE: emulated TPM) is the problem here though.

Eventually dracut times out and drops into a rescue shell in the initrd. The cryptsetup unit faied with a Current policy digest does not match stored policy digest, cancelling TPM2 authentication attempt. error. Further, an error of No passphrase or recovery key registered is also printed. I don't think this is a PCR issue.

Some observations:

• The latter of these issues (lack of failover way to unlock) is most certainly a bug in this install path. If the means for unlocking the LUKS volume will be the TPM, a recovery key must be set to allow the system to boot in the event PCRs change. Alternatively, allow the user to provide a normal password. One or both of these failover methods needs to be supported at install time (not after).
• It is not clear from the documentation or CLI interface what PCRs the volume gets bound to in the TPM. The defaults should be documented and also user configurable. Right now based on reading the source, it looks like the LUKS volume binds to no PCR so it will always unlock as long as the TPM is present PCR 7, the systemd-cryptenroll default.

[jmpolom]

I am able to unlock the LUKS volume via TPM on a live system (FCOS) booted on the same VM I used to perform the bootc install to-disk --block-setup tpm2-luks from. It looks like the TPM2 binding on the volume is valid.

I think my proposal is if it remains possible to install to a block device and into a LUKS volume, a user supplied password ought to be an option (it appears this would be a straightforward modification). The LUKS password can be changed or removed later on once the system is installed. If a user opts in to foregoing a password by not specifying one, then you would get the current behavior with no means of recovery. This would necessitate dropping the headless karg which should not be default either as it could severely hinder developmental debugging. Perhaps a second improvement would be to support a recovery key method but I see that as completely secondary to supporting a means to setup the volume with a plain ol password.

[jmpolom]

Looks like this was observed with bootc 0.1.7 on an image created from the treefile configuration at this point that resulted in this container image

[cgwalters]

Thanks for filing this, indeed we have a CI gap on this here.

(The integration of bootc install needs to be at least partly owned by the particular OS you're using; we will try to be generic where we can of course).

[jmpolom]

There needs to be a failover way to unlock any LUKS volumes bound to tokens at install time. Right now bootc actually creates a temp password and then discards it. At a very basic level this could just be output to the console as a very crude method to provide a way in if the TPM method isn't working. This is a basic issue with the to-disk install path. There really should be some user configurable options though.

Unless the thought is bootc will only handle the bottom half of the logic (installing to a pre-made filesystem) and to-disk install path gets removed??

The integration of bootc install needs to be at least partly owned by the particular OS you're using

This was tested with Fedora 39 which has plenty of ostree based releases so I don't think that is necessarily the issue. Are there specific additional integrations needed (link to docs please)?

Based on my detailed and exhaustive review of the existing state of the art (IE: everything here), I'm really coming up short for why this specific configuration isn't working. Looking at how the LUKS configuration is passed to the initrd via kargs and my double checking that the necessary libs are present in the initrd, the failure to unlock really doesn't make sense.

Are any modifications beyond adding a few dracut modules needed to support TPM2 based unlocking with systemd-cryptsetup in the initrd? This is about what I'd expect to need to do to preconfigure an initrd with the proper libs. I've done this before on Debian and prior versions of Fedora successfully.

Hazard a guess at what else should be checked here?

[cgwalters]

There needs to be a failover way to unlock any LUKS volumes bound to tokens at install time.

For some use cases, all data on disk is effectively a cache, and having a failover adds risk and management overhead - questions around how is the failover secret rotated, etc.

Unless the thought is bootc will only handle the bottom half of the logic (installing to a pre-made filesystem) and to-disk install path gets removed??

There is some strong inherent tension here but the overall thought on the design is that to-disk is designed for the simple cases (obviously, "plain filesystem" is very simple), and the current tpm2-luks is arguably on the bounds of "simple" - and you can use to-filesystem to set things up however you want for all the other cases.

[jmpolom]

There needs to be a failover way to unlock any LUKS volumes bound to tokens at install time.

For some use cases, all data on disk is effectively a cache, and having a failover adds risk and management overhead - questions around how is the failover secret rotated, etc.

Valid concern for that specific case but that has got to be one hell of an edge case. Even if that is a main use case it creates a situation that is so unnecessarily difficult to debug that there needs to be an alternative to enable basic development. Such behavior that discards/skips/doesn't support a manual failover mechanism needs to be explicitly opted into by the user. Or the silly default must have an opt out. Right now it is forced opt in which is a problem.

and the current tpm2-luks is arguably on the bounds of "simple"

I'd encourage you to review all the logic in the supporting libraries and systemd itself for this to work. It is anything but simple and it increases the dependency footprint substantially (proper libs need to end up in the initrd, for instance). Far more complex than a password.

[jmpolom]

Tried to bootc install to-disk --block-setup tpm2-luks with the latest centos-bootc fedora-eln image (sha256: 174fb00e242e7aaa2d9c5f34056caea7fd726433949c0dedd12158aa5e6b1d0f) and it fails to unlock the root volume upon reboot. Failed systemd-cryptsetup unit trying to unlock via TPM2.

I tried the build from last week previously, and it successfully unlocked the LUKS root volume on reboot (sha256:1c5e91ab395665ca11e2c1a17df18beec39d63fd2948f15add9fd95e45c0c85b). Clearly there have been some package changes in the past few days that broke this??

While waiting for this update, I also tried both F39 and F40/next based builds with the latest bootc from the copr. Same story with those -- failed systemd-cryptsetup units trying to unlock via TPM.

[cgwalters]

Thanks Jon, this is very valid feedback and thanks for looking at this.

Valid concern for that specific case but that has got to be one hell of an edge case.

For example in cloud environments I may want to enable LUKS to be very sure my data is encrypted, and binding to the virtualized TPM2 is a generic baseline for that that at least helps ensure that if e.g. someone gets access somehow to an underlying block store they can't read the data. I am sure some people want a fallback password even in cloud, but it's not very "cloud native" to log in interactively on a console in an IaaS.

Even if that is a main use case it creates a situation that is so unnecessarily difficult to debug that there needs to be an alternative to enable basic development. Such behavior that discards/skips/doesn't support a manual failover mechanism needs to be explicitly opted into by the user.

Yes, fair enough.

OK so I think what my inclination here is to make tpm2-luks require a flag in the install config in the container image. This way, the OS container image creator must opt-in to saying they support it (e.g. they have the required components in the initramfs, etc.)

Then in parallel of course, we should:

• Make this work
• Expand on doing things like this via bootc install to-filesystem to be clear that arbitrarily complex storage setups can be owned by codebases other than bootc

[jmpolom]

For example in cloud environments I may want to enable LUKS to be very sure my data is encrypted, and binding to the virtualized TPM2 is a generic baseline for that that at least helps ensure that if e.g. someone gets access somehow to an underlying block store they can't read the data. I am sure some people want a fallback password even in cloud, but it's not very "cloud native" to log in interactively on a console in an IaaS.

A great use for the recovery key option. If bootc could cleanly output the recovery key a secondary process could store it in a secrets vault. You really cannot have a TPM2 only binding. Even systems like Windows will provide recovery keys for when the PCRs change (as they are designed to do). Some deployments will want to bind to a lot more than PCR 7 and some of those PCRs may change even on OS update.

OK so I think what my inclination here is to make tpm2-luks require a flag in the install config in the container image. This way, the OS container image creator must opt-in to saying they support it (e.g. they have the required components in the initramfs, etc.)

There would also need to be a flag that could either have the system create and provide at install time a temp password or enroll a recovery key. Ideally both should be supported (recovery keys are lengthy and would be particularly obnoxious to deal with for repetitive early stage testing) at the user or image builders discretion.

Specifically as to the initramfs components, the requirements need to be documented somewhere. Just having other works (like the centos-bootc) to reference isn't a great experience. It isn't entirely straightforward exactly what dracut modules should be added in. It's a bit different configuring the initramfs here because the image is being composed off board from the system it will run on, so any auto detection of things isn't going to work.

Expand on doing things like this via bootc install to-filesystem to be clear that arbitrarily complex storage setups can be owned by codebases other than bootc

Personally I would perhaps consider just removing the to-disk workflow. I'm not sure what value it brings if it can only handle trivially simple deployments. Is the juice from the added complexity actually worth it? I think it might be hard to avoid feature/complexity creep on what looks like a bare metal installer feature.

Might be easier to just document a bit better how to use to-filesystem with an external workflow to prepare the disks and have a reference shell script that can be included in an image build to do this. Something to think about.

[cgwalters]

#445 will effectively turn this off by default for now.

Personally I would perhaps consider just removing the to-disk workflow.

I find it extremely useful as it provides a generic baseline, allowing a container image to self-install onto a block device without any other externally versioned infrastructure. (It also tries hard to force configuration to come from the container image by default).

Now, I did also file #440 which would make it much easier for containers to configure things in arbitrary ways.