Ceph Kernel Module Missing.

[ajh34]

modprobe: FATAL: Module ceph not found in directory /lib/modules/4.14.98-v7+

Looks like the Ceph package does not install the required kernel module to mount cephfs filesystems. Same issue on stretch and previous release.

[JamesH65]

@ajh34 We do not enable CEPH in our kernel builds. It seems like a fairly niche system, and unless its impact on memory and CPU bandwidth is minimal (close to zero), we would be unlikely to incorporate it by default. If you can provide those figures, it would give us something to consider. It seems likely that as a module it impact would indeed be minimal, but we would need to be sure.

[imjustmatthew]

Unexpectedly ran into this same issue. I would never have expected a modern kernel didn't have Cpeh included. The performance impact of this as a module should be extremely small when it's not loaded and laoding of the module will only occur when a user actually uses it to mount a filesystem. There is a Ceph-FUSE option, but that is much lower performing that the kernel module.

[JamesH65]

@pelwell @popcornmix Any thoughts?

[pelwell]

As a filesystem I would expect no impact on performance if it isn't used, but there could be some overhead on the static (non-module) kernel size. The usual expectation on somebody asking for a feature to be added is for them to do a trial build and show the size of any additional modules, and the size of the kernel .img and free memory when it is used, along with the same information for a kernel that differs only in that the feature is not enabled.

[StrayLightning]

I built from fe915de, following the instructions here: https://www.raspberrypi.org/documentation/linux/kernel/building.md

The resultant image:
Linux cluster0 4.19.127-v7l-ceph+ #1 SMP Thu Jul 9 15:18:05 BST 2020 armv7l GNU/Linux

The kernel previously:
Linux cluster1 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux

(Different hostnames, but the nodes were updated at the same time.)

The resultant kernel images:
-rwxr-xr-x 1 root root 5801056 Jul 6 17:13 kernel7l.img
-rwxr-xr-x 1 root root 5777480 Jul 9 18:05 kernel-ceph.img

Of course, this is not an apples-to-apples comparison: If I get a chance I'll run a build from the same point that the released kernel was built from.

Changes to the kernel configuration:

21c21
< CONFIG_LOCALVERSION="-v7l"
---
> CONFIG_LOCALVERSION="-v7l-ceph"
1604c1604,1606
< # CONFIG_CEPH_LIB is not set
---
> CONFIG_CEPH_LIB=m
> # CONFIG_CEPH_LIB_PRETTYDEBUG is not set
> # CONFIG_CEPH_LIB_USE_DNS_RESOLVER is not set
6478c6480,6482
< # CONFIG_CEPH_FS is not set
---
> CONFIG_CEPH_FS=m
> # CONFIG_CEPH_FSCACHE is not set
> # CONFIG_CEPH_FS_POSIX_ACL is not set

[StrayLightning]

Using the modified config and building from the tag raspberrypi-kernel_1.20200601-1:

-rwxr-xr-x 1 root root 5776216 Jul 9 21:50 kernel-ceph.img

Linux cluster0 4.19.118-v7l-ceph+ #2 SMP Thu Jul 9 21:45:01 BST 2020 armv7l GNU/Linux

And this is the official build again for comparison:

-rwxr-xr-x 1 root root 5801056 Jul 6 17:13 kernel7l.img

For clarification, the configuration was modified by explicitly setting:

CONFIG_CEPH_LIB=m
CONFIG_CEPH_FS=m

and then choosing the default options for the other options that the build prompts for.

[StrayLightning]

Sorry, I forgot to include the sizes of the modules:

-rw-r--r-- 1 root root 299404 Jul 9 21:50 /lib/modules/4.19.118-v7l-ceph+/kernel/fs/ceph/ceph.ko
-rw-r--r-- 1 root root 302620 Jul 9 21:50 /lib/modules/4.19.118-v7l-ceph+/kernel/net/ceph/libceph.ko

[pelwell]

Please explain why we should tax all Pi users 4 * 600kB = 2.4MB for this feature.

[StrayLightning]

I'm not saying you should :-) I happened to be building the Ceph modules to try them out, and noticed this issue with the last comment being a request for the information.

I do notice, however, that the change in disk space for the kernel modules under /lib/modules is an increase of 0.14% between the supplied v7l directory and my build.

[StrayLightning]

Interestingly, I notice that there's just shy of 8MiB of kernel modules for OCFS2 (https://en.wikipedia.org/wiki/OCFS2) support in the current Raspberry Pi OS image. Would that be because of this:

https://blogs.oracle.com/developers/building-the-world%e2%80%99s-largest-raspberry-pi-cluster

[StrayLightning]

Add to that another 2MiB for GFS2 (https://en.wikipedia.org/wiki/GFS2) support and 2MiB plus change for DRBD (https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device) support.

At this point it is looking a bit strange that the Ceph modules are not included.

[cawolf]

I also ran into this issue trying to set up a local ceph cluster in k3s. As I do not know too much about kernel modules: is there a way that you compile the module, but do not load it by default until a user opts in to load it?

[SnipGhost]

At this point it is looking a bit strange that the Ceph modules are not included.

Yes, I think the support of the ceph is extremely necessary. Considering current trends - many people want to try production-ready K8S with Ceph distributed storage at home. Many companies use this technology stack, I conduct experiments on the implementation of the latest technologies on my raspberry cluster, in order to then apply this experience in real infrastructure at work.

@pelwell, rebuilding the kernel is tedious every time, especially since not everyone is ready to dive into it. IMHO, 2.4MiB is not such a price that it somehow affects ordinary users...

[cawolf]

Just to let you know: missing out on this lead me to switch to pure debian images as my base image. Setting up ceph in this setup was a breeze.

[kristvanbesien]

Of course one can go back to using the upstream kernel (or a distro that used it). But my usage case is one where I build a K8s cluster on RPI4 that are POE powered. The POE hat requires the downstream kernel, or the fans do not spin up (and then the systems overheat), however running Rook/Ceph on top of k8s rquires RBD...

[jfcoz]

Any news on this ? The missing Ceph support forbids the usage of Rook on K3S.

[insanemal]

What's the holdup on this RBD and CephFS are used widely with K3s and other K8s solutions. At the moment I need to either install a non-raspbian distro or rebuild the kernel to get ceph support for k8s.

This is silly ESPECIALLY when you have other much larger modules enabled for other "show boat" usages. Also, on what planet is 2MB an issue when you recommend a minimum of 8GB for the SD card...

Moreover you can always delete modules after install if the 2MB is so critical. Or, and here's a wild idea, package up kernel modules as their own package for ceph, OCFS2, GFS2, and DRBD. So build em, package them separately. Because @pelwell I want my 10+MB back from OCFS2, GFS2, and DRBD. I don't know why we tax every RPi install 12MB for stuff most people arern't going to use. I mean who in their right mind would use RPi's for DRBD?

[JeremyBolster]

@pelwell @JamesH65
Is this being considered? If not, is there a process to have this feature considered?
Kernel size impact has been shown to be minimal (2MB of a recommended 8GB SD card or 0.025%); on par with other storage modules that are included in the raspberrypi kernel.

[PhilippKuntschik]

If I remember correct, there was the option to install additional kernel-modules with apt install linux-modules-extra-raspi. Would this be a possibility to get kernel modules without "taxing all Pi users"?

Also, with the turing-pi having achieved >7000 bakers, I assume there will be a significant demand for this feature coming in later this year.

[pelwell]

The bad news is that since the last size measurement (on 4.19), the module size has increased:

375136 Jul 21 11:17 ./fs/ceph/ceph.ko
382536 Jul 21 11:17 ./net/ceph/libceph.ko

This takes the overhead to ~4*750kB = ~3MB.

However, the good news is that a few months ago we enabled module compression, so when installed the sizes drop considerably:

-rw-rw-r-- 1 root root 117804 Jul 21 11:20 ./fs/ceph/ceph.ko.xz
-rw-rw-r-- 1 root root 113524 Jul 21 11:20 ./net/ceph/libceph.ko.xz

I think this is acceptable for a (currently) niche feature, therefore the option has been added to all current kernel branches and will be in future releases.

[pygmymarmoset]

Can rbd be added as well? I think almost anywhere that ceph is wanted, rbd will be wanted as well, including the rook/ceph stuff in kubernetes.

CONFIG_BLK_DEV_RBD=m

-rw-r--r-- 1 root root 36164 Aug 30 23:18 /lib/modules/5.15.61-v8+/kernel/drivers/block/rbd.ko.xz

[DavidZidar]

Will this change include ceph-fuse as well?

[pelwell]

Don't add further requests onto existing issues - they will be ignored.