IPv6 loadbalancing with keepalived/IPVS is not possible due to missing CONFIG_IP_VS_IPV6 kernel config

[chillout2k]

Describe the bug
It´s not possible to set up IPv6 capable IP virtual servers (IPVS).

An Ubuntu related issue can be found here

To reproduce
Install ipvsadm and try to add a trivial IPv6-IPVS:

ipvsadm -A -t [fc00::1]:80
ipvsadm -a -t [fc00::1]:80 -r [fc00::3]:80 -g
Resource temporarily unavailable

AFAIK the kernel config attribute CONFIG_IP_VS_IPV6 must be enabled, but it´s not. To get the current kernel config run this:

modprobe configs
zgrep CONFIG_IP_VS_IPV6 /proc/config.gz 
# CONFIG_IP_VS_IPV6 is not set

Expected behaviour
Setup of IPv6 based IPVS is possible ;)

Actual behaviour
Setup of IPv6 based IPVS is not possible ;)

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
  Pi1, Pi3, Pi3B+

• Which OS and version (cat /etc/rpi-issue)?
  Raspberry Pi reference 2017-03-02
  Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, f563e32202fad7180c9058dc3ad70bfb7c09f0fb, stage2

• Which firmware version (vcgencmd version)?
  Nov 4 2018 16:32:03
  Copyright (c) 2012 Broadcom
  version ed5baf9520a3c4ca82ba38594b898f0c0446da66 (clean) (release)

• Which kernel version (uname -a)?
  4.14.79-v7+ USB-Harddrive gets fully powered down #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux
  4.14.94-v7+ [4.2] fix cyclic dma setup and i2s parameters #1193 SMP Tue Jan 22 15:34:30 GMT 2019 armv7l GNU/Linux

Logs
Not needed from my point of view

Additional context
Relevant for IPv6 load balancing with Docker/LXD

Thank you very much in advance for your attantion and help.
Dominik

[chillout2k]

Please have a look at #265 (from 2013) regarding IPv4 based IPVS

[JamesH65]

Do you have any figures for impact on kernel size and performance? Given a limited resources device like the Pi we like to ensure that kernel changes for smaller use cases do not have a detrimental impact.

[PKizzle]

Is there any update on the kernel size / performance impact?

[JamesH65]

This is not something we would do, it's down to the requestor. Since they have not responded for over two years, I guess it will need to be done by someone else.

[edwinbalani]

I guess it will need to be done by someone else.

👋

I stumbled on this issue by chance today -- I'm keen to have IPVS IPv6 support in the raspberrypi kernel to avoid switching kernels wholesale (or building my own) for what I'm trying to achieve, so hopefully I can be the someone else here.

ip_vs and ip_vs_* are compiled as modules already, so I'm not sure how impactful any noticeable change in performance would actually be. For the majority of running kernels, IPVS will never be loaded or set up in the first place, so no chance of a performance impact there. By the same token, it being in a .ko will mean no impact on /boot space, only on the root partition that's probably much bigger.

Regardless I will see if I can compile the ip_vs modules on the latest raspberrypi/linux 5.10 tag for arm64 with and without the option set, and report back on size change.

[edwinbalani]

I will see if I can compile the ip_vs modules on the latest raspberrypi/linux 5.10 tag for arm64

For clarity, I chose these because that's what I'm running, but I can do for armhf as well

[edwinbalani]

As a rough and ready indication, I compiled the modules with this script:

#!/bin/sh
set -e

make_rpi () { make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- "$@"; }
build_modules() {
  make_rpi -j8 \
        net/netfilter/ipvs/ip_vs_app.ko \
        net/netfilter/ipvs/ip_vs_conn.ko \
        net/netfilter/ipvs/ip_vs_core.ko \
        net/netfilter/ipvs/ip_vs_ctl.ko \
        net/netfilter/ipvs/ip_vs_dh.ko \
        net/netfilter/ipvs/ip_vs_est.ko \
        net/netfilter/ipvs/ip_vs_ftp.ko \
        net/netfilter/ipvs/ip_vs_lblc.ko \
        net/netfilter/ipvs/ip_vs_lblcr.ko \
        net/netfilter/ipvs/ip_vs_lc.ko \
        net/netfilter/ipvs/ip_vs_nfct.ko \
        net/netfilter/ipvs/ip_vs_nq.ko \
        net/netfilter/ipvs/ip_vs_pe.ko \
        net/netfilter/ipvs/ip_vs_pe_sip.ko \
        net/netfilter/ipvs/ip_vs_proto_ah_esp.ko \
        net/netfilter/ipvs/ip_vs_proto.ko \
        net/netfilter/ipvs/ip_vs_proto_sctp.ko \
        net/netfilter/ipvs/ip_vs_proto_tcp.ko \
        net/netfilter/ipvs/ip_vs_proto_udp.ko \
        net/netfilter/ipvs/ip_vs_rr.ko \
        net/netfilter/ipvs/ip_vs_sched.ko \
        net/netfilter/ipvs/ip_vs_sed.ko \
        net/netfilter/ipvs/ip_vs_sh.ko \
        net/netfilter/ipvs/ip_vs_sync.ko \
        net/netfilter/ipvs/ip_vs_wlc.ko \
        net/netfilter/ipvs/ip_vs_wrr.ko \
        net/netfilter/ipvs/ip_vs_xmit.ko \
        net/netfilter/xt_ipvs.ko
}
count_ko_bytes () { find . -name '*.ko' -exec wc -c {} + ; }

## Main script
make_rpi clean
make_rpi bcmrpi3_defconfig
build_modules; count_ko_bytes > module-sizes-original.txt

make_rpi clean
sed -i -e 's/^# CONFIG_IP_VS_IPV6 is not set/CONFIG_IP_VS_IPV6=y/' .config
build_modules; count_ko_bytes > module-sizes-with-ipv6.txt

I identified the modules that are affected just by searching for CONFIG_IP_VS_IPV6. All the source files listed above were hits, as well as include/net/ip_vs.h -- but that itself is included only in the above modules, so there are no other references that might impact things like struct sizes.

The result is about 20 kB more code. Most of this (15 kB) is gained in ip_vs_xmit, which adds IPv6 implementations of the various "packet transmitters" in IPVS. Not much on the scale of things? Here is the difference I got per module, and in the total, all in bytes:

--- module-sizes-original.txt   2022-01-31 23:08:35.907569496 +0000
+++ module-sizes-with-ipv6.txt   2022-01-31 23:14:58.261511552 +0000
@@ -1,29 +1,29 @@
  13704 ./net/netfilter/ipvs/ip_vs_app.ko
- 40088 ./net/netfilter/ipvs/ip_vs_conn.ko
+ 42784 ./net/netfilter/ipvs/ip_vs_conn.ko
- 40240 ./net/netfilter/ipvs/ip_vs_core.ko
+ 45944 ./net/netfilter/ipvs/ip_vs_core.ko
- 74488 ./net/netfilter/ipvs/ip_vs_ctl.ko
+ 76960 ./net/netfilter/ipvs/ip_vs_ctl.ko
-  8088 ./net/netfilter/ipvs/ip_vs_dh.ko
+  8152 ./net/netfilter/ipvs/ip_vs_dh.ko
   6440 ./net/netfilter/ipvs/ip_vs_est.ko
  14520 ./net/netfilter/ipvs/ip_vs_ftp.ko
- 13432 ./net/netfilter/ipvs/ip_vs_lblc.ko
+ 13688 ./net/netfilter/ipvs/ip_vs_lblc.ko
- 15160 ./net/netfilter/ipvs/ip_vs_lblcr.ko
+ 15416 ./net/netfilter/ipvs/ip_vs_lblcr.ko
   5880 ./net/netfilter/ipvs/ip_vs_lc.ko
   8768 ./net/netfilter/ipvs/ip_vs_nfct.ko
   5944 ./net/netfilter/ipvs/ip_vs_nq.ko
   7744 ./net/netfilter/ipvs/ip_vs_pe.ko
-  7800 ./net/netfilter/ipvs/ip_vs_pe_sip.ko
+  8408 ./net/netfilter/ipvs/ip_vs_pe_sip.ko
- 11392 ./net/netfilter/ipvs/ip_vs_proto.ko
+ 12320 ./net/netfilter/ipvs/ip_vs_proto.ko
   5736 ./net/netfilter/ipvs/ip_vs_proto_ah_esp.ko
- 14072 ./net/netfilter/ipvs/ip_vs_proto_sctp.ko
+ 14200 ./net/netfilter/ipvs/ip_vs_proto_sctp.ko
- 15360 ./net/netfilter/ipvs/ip_vs_proto_tcp.ko
+ 16336 ./net/netfilter/ipvs/ip_vs_proto_tcp.ko
- 11760 ./net/netfilter/ipvs/ip_vs_proto_udp.ko
+ 12672 ./net/netfilter/ipvs/ip_vs_proto_udp.ko
   6544 ./net/netfilter/ipvs/ip_vs_rr.ko
- 10016 ./net/netfilter/ipvs/ip_vs_sched.ko
+ 10200 ./net/netfilter/ipvs/ip_vs_sched.ko
   5952 ./net/netfilter/ipvs/ip_vs_sed.ko
-  8776 ./net/netfilter/ipvs/ip_vs_sh.ko
+  8840 ./net/netfilter/ipvs/ip_vs_sh.ko
- 28120 ./net/netfilter/ipvs/ip_vs_sync.ko
+ 28856 ./net/netfilter/ipvs/ip_vs_sync.ko
   5952 ./net/netfilter/ipvs/ip_vs_wlc.ko
   7752 ./net/netfilter/ipvs/ip_vs_wrr.ko
- 19536 ./net/netfilter/ipvs/ip_vs_xmit.ko
+ 34384 ./net/netfilter/ipvs/ip_vs_xmit.ko
-  8088 ./net/netfilter/xt_ipvs.ko
+  8624 ./net/netfilter/xt_ipvs.ko
-421352 total
+452720 total

Built from 650082a (tag 1.20220120, 1.20220118)
Compiler version: aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0 (Debian buster amd64)

[PKizzle]

@JamesH65 Is there anything else you need regarding external input?

[JamesH65]

Not my domain, @pelwell on the other hand...

[pelwell]

If this is such a useful option, why do only a handful of MIPS defconfigs specify it?

[PKizzle]

I can honestly only speculate why this option is not enabled in that many MIPS defconfigs. While IPVS load-balancing was added to Kubernetes in 2018, IPv6 support (dual-stack) has been added very recently in 2021. So, I guess that the interest just wasn't there before Kubernetes added this functionality. But why not be one of the first to offer the Kubernetes community a pre-built kernel with that option enabled by default?
These are the benefits of using IPVS over IPTABLES in Kubernetes: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md#ipvs-vs-iptables

[PKizzle]

@pelwell Do you need any further information?

[pelwell]

No, I just needed some time and sufficient motivation. We have to way the benefits of new features against the costs to all the users who don't use those features. In this case, although the feature is niche, the changes are to modules (meaning the core kernel doesn't increase in size or run slower), and the size increase is relatively small (especially with module compression enabled).

The config change has been applied to all the current branches, and the updated modules will be in future kernel releases.