debos.yaml sequence of commands causing ext4.rootfs corruption

[nuclearcat]

Current kernelci-core git builds rootfs with consistently broken rootfs.ext4 (mounts with errors, some content missing, etc), several rootfs.ext4 builds on storage.kernelci.org i verified have "needs_recovery" flag, which indicate unclean mount and possible corruption.

In my test cases, in some circumstances it is severe (rootfs not bootable). The most annoying thing is that this bug can cause random, hard to trace filesystem corruption.

In "config/rootfs/debos/rootfs.yaml" after filesystem-deploy there is "Move image to output directory" and compression by xz.
While in debos "actions/filesystem_deploy_action.go" it is said "After this action has ran, subsequent actions are executed on the mounted output image." and we are moving image, then compressing it, while it is not unmounted.
Looking to debos code, probably it will get unmounted only after rootfs.yaml is completed.

Just to verify, when i just comment move and xz actions in rootfs.yaml - ext4 image is consistent.

To reproduce/verify:

wget http://storage.kernelci.org/images/rootfs/debian/buster-igt/20211105.0/amd64/rootfs.ext4.xz
xz -d rootfs.ext4.xz
losetup -f -P rootfs.ext4
dumpe2fs /dev/loop0p1 | grep needs_recovery
# Here it will do recovery
mount /dev/loop0p1 /mnt 
umount /mnt
# Now we can see it is clean, if it is not broken beyond repair
dumpe2fs /dev/loop0p1 | grep needs_recovery
losetup -d0

As I understand, the easiest solution is to move the mv and xzip operations out of rootfs.ext4, and execute after it.
If you think this is the bug, and proposed solution is ok, i can make a PR request.

P.S. Maybe worth to discuss in IRC, my nickname is nuclearcatlb

[sjoerdsimons]

You shouldn't have to move the operation out of the rootfs.yaml but they should be executed with postprocess: true which will guarantee it runs after the filesystem has been unmounted etc. Any processing of the image file debos generates without setting that flag will cause undefined behaviour for reasons you mentioned above (image is still mounted, not everything may be written back to it yet etc).

While those action run last by definition, so you don't have to specifically move them to the end of the recipe, for easy of understanding that's probably recommendable anyway :).

[nuclearcat]

You shouldn't have to move the operation out of the rootfs.yaml but they should be executed with postprocess: true which will guarantee it runs after the filesystem has been unmounted etc. Any processing of the image file debos generates without setting that flag will cause undefined behaviour for reasons you mentioned above (image is still mounted, not everything may be written back to it yet etc).

While those action run last by definition, so you don't have to specifically move them to the end of the recipe, for easy of understanding that's probably recommendable anyway :).

You are right, just tested it, postprocess: true fixes issue.
I guess i should prepare PR, if it is ok for project maintainers.

[gctucker]

Thanks for catching this issue and sending a fix.

It's true that ext4 images aren't used at all for kernelci.org as all the Debian jobs currently use the initrd + plain tarball over NFS. We may start using ext4 at some point by flashing the images on persistent storage, or maybe for some virtual devices that can mount the ext4 images directly.