cpuset disabled

[rcwoolley]

I noticed that cpuset was disabled when using my Raspberry Pi. Others have also run into this in the forums: https://www.raspberrypi.org/forums/viewtopic.php?t=178981&p=1139830

I reverted this commit and it restored cpuset: 7dbc189

It seems that the commit was actually trying to disable the memory resource controller. I can confirm that "memory" remains enabled when the patch is present. Also using cgroup_enable=cpuset does re-enable
cpuset.

I thought I should report it in case it helps others. Would you be open to reverting the commit?

[popcornmix]

Back when this commit was first introduced:
CONFIG_MEMCG and CONFIG_CGROUP_MEM_RES_CTLR
resulted in 8 + 24 bytes of accounting memory per 4K page.

http://cateee.net/lkddb/web-lkddb/MEMCG.html
Note that setting this option increases fixed memory overhead associated with each page of memory in the system. By this, 8(16)bytes/PAGE_SIZE on 32(64)bit system will be occupied by memory usage tracking struct at boot. Total amount of this is printed out at boot.
Only enable when you're ok with these trade offs and really sure you need the memory resource controller. Even when you enable this, you can set "cgroup_disable=memory" at your boot option to disable memory resource controller and you can avoid overheads. (and lose benefits of memory resource controller)
http://cateee.net/lkddb/web-lkddb/CGROUP_MEM_RES_CTLR.html
Provides a memory resource controller that manages both anonymous memory and page cache. (See Documentation/cgroups/memory.txt)
Note that setting this option increases fixed memory overhead associated with each page of memory in the system. By this, 20(40)bytes/PAGE_SIZE on 32(64)bit system will be occupied by memory usage tracking struct at boot. Total amount of this is printed out at boot.
Only enable when you're ok with these trade offs and really sure you need the memory resource controller. Even when you enable this, you can set "cgroup_disable=memory" at your boot option to disable memory resource controller and you can avoid overheads. (and lose benefits of memory resource controller)
This config option also selects MM_OWNER config option, which could in turn add some fork/exit overhead.

So we made the choice a boottime switch which avoided the memory overhead unless you added cgroup_enable=memory to cmdline.txt to enable it (upstream code only provides a parameter to disable group options).

Now, it seems the descriptions of the config options have changed, and I haven't tracked down what the current cost is of enabling the options.

What exactly is your issue? Are you wanting to use memory groups? Does cgroup_enable=memory make them work?

[rcwoolley]

My issue is that I want to use the cpuset functionality in cgroups and this commit is disabling the cpuset subsystem when it means to disable memory.

I suspect that the cgroup subsystems don't get initiated in a strict order, which would cause this line:
+static u16 cgroup_disable_mask __initdata = 1<<0;
to inadvertently disable the cpuset controller instead of the memory controller.

I searched Google for "raspberrypi dmesg cgroup" and saw a number of examples where the printks "Initializing cgroup subsys ****" are printed in different orders. Generally, "Initializing cgroup subsys cpuset" seemed to be listed first, however that wasn't always the case.

[popcornmix]

The initial version of the patch (0ce4380) was safer as the disable flags were in separate structures.

Would you be happy if 7dbc189 were reverted and CONFIG_MEMCG was removed?

CONFIG_MEMCG was mainly added for the Epiphany browser, but as that is no longer the recommended browser, it is of less interest to enable it.

[rcwoolley]

I can't recommend disabling CONFIG_MEMCG as I don't know how long it has been enabled for in the official kernels people are using on their RPis. Someone else may depend on it. However, reverting the patch would allow people to use cpuset set again, which is unlikely to negatively affect anyone.

[popcornmix]

I've checked and CONFIG_MEMCG enabled seems to cost ~2MB on a freshly booted Pi3.
Disabled:

pi@domnfs:~ $ free -h
             total       used       free     shared    buffers     cached
Mem:          590M        86M       503M       4.8M        52K        46M
-/+ buffers/cache:        39M       550M
Swap:         511M         0B       511M

Enabled:

pi@domnfs:~ $ cat free1 
             total       used       free     shared    buffers     cached
Mem:          590M        88M       501M       4.8M        52K        49M
-/+ buffers/cache:        39M       550M
Swap:         511M         0B       511M

[JamesH65]

@rcwolley @popcornmix Will be closing if no further comments are posted.

[rcwoolley]

Hi @JamesH65,

Based on this line (https://github.com/raspberrypi/linux/blob/rpi-4.9.y/kernel/cgroup.c#L5640 ) it looks as if the patch is still present and potentially disabling cpuset by mistake.

It is my suggestion to revert the patch to restore cpuset.

Regards,
Rob

[JamesH65]

@popcornmix @pelwell Any thoughts? I have no ideas what this should be.

[pelwell]

It looks like we need an alternative way of implementing an implicit cgroup_disable=memory that copes with different bit assignments. Rather than adding a generic cgroup_enable= parameter (which is only really useful for the the memory subsystem), we could add a flag to enable the memory cgroup which is initially cleared, and then one of the following:

1. In the for_each_subsys loop in cgroup_init_early, if the flag is not set then compare ss->name with "memory" and set the corresponding mask bit if it matches.

2. Somewhere near the start of cgroup_init, call cgroup_disable("memory"); if the flag is not set.

If that sounds reasonable then I'll work it up into a patch.

[popcornmix]

Sounds fine to me.

[pelwell]

See #2201

[rcwoolley]

Thanks, Phil. This will make it much easier to use cpuset on the official Raspberry foundation kernels.

[pelwell]

Is that confirmation that it works for you?

[rcwoolley]

I'll take the action to compile a fresh kernel and confirm it works later today.

[rcwoolley]

Sorry for the delay. I cloned the latest changed and confirmed with /proc/cgroups that memory was disabled by default and enabled when I set the kernel param.

I did see a warning about not checking the return value of kstrtobool. I don't think it changes the behaviour.

Thanks for making the change!

[pelwell]

After some feedback and a rethink I've reworked this code as two new patches. The first makes cgroup_enable=memory work as expected, and the second adds the cgroup_memory= flag as a backwards compatibility patch which may be dropped for 4.14.