How are container pid limits chosen for --pids-limit 0?

[jjenne-cisco]

I expect that the container's pids.max should be set to 'max', but this is only true for Cgroups v2. No for Cgroups v1.

Here are some datapoints on different VMs with cgroups v1 (ubuntu 20.04, podman 3.4.2) and cgroups v2 (ubuntu 22.04, podman 3.4.4):

root@cgroupsv1:~$ docker run ubuntu cat /sys/fs/cgroup/pids/pids.max
max
root@cgroupsv1:~$ podman run ubuntu cat /sys/fs/cgroup/pids/pids.max
2048
root@cgroupsv1:~$ podman run --pids-limit 0 ubuntu cat /sys/fs/cgroup/pids/pids.max
7087
root@cgroupsv1:~$ podman run --pids-limit 0 --runtime /usr/sbin/runc ubuntu cat /sys/fs/cgroup/pids/pids.max
7087

root@cgroupsv2:~$ podman run --pids-limit 0 ubuntu cat /sys/fs/cgroup/pids.max
max
root@cgroupsv2:~$ podman run --pids-limit 0 --runtime /usr/sbin/runc ubuntu cat /sys/fs/cgroup/pids.max
7076

(Runtime comparisons for extra data points)

Firstly, I'm not exactly sure where the "unlimited" numbers are taken from as on my VMs /sys/fs/cgroup/{machine,system,user}.slice/pids.max are set to 'max'. Nowhere in the cgroup v1 hierarchy should podman be inheriting the above numbers.

I know that the default pid limit, when unlimited is not set on the CLI, is 2048 as set here and I can see that the pids.max file is being written with the limit here. But I can't work out the root of where the "unlimited" values are taken from and why podman run --pids-limit 0 ubuntu cat /sys/fs/cgroup/pids/pids.max is not 'max' like docker's default.

Does anyone know any more about how or where the resources.Pids.Limit gets set in the case of --pids-limit 0?

---

EDIT: There is an e2e test with the line "pids-limit not supported on cgroup V1" - I'm assuming this is an incorrect skip reason as --pids-limit seems in every way supported on cgroups v1.

[mheon]

@giuseppe PTAL

[giuseppe]

if you run rootless, the cgroups are not configured for the container, so it will inherit the cgroup from Podman.

[rhatdan]

Does this only work on cgroupv1?

[jjenne-cisco]

Yes, the datapoints above are all on a cgroupsv1 host. On cgroupsv2 I see "max" like I'd expect:

root@cgroupsv2:~$ podman run --pids-limit 0 ubuntu cat /sys/fs/cgroup/pids.max
max

[jjenne-cisco]

On trying docker + systemd cgroup driver, it looks like this is related to cgroupsv1 and systemd cgroup manager:

root@ubuntu:~# cat /etc/docker/daemon.json
{
  "ipv6": true,
  "fixed-cidr-v6": "fd00::/80"
}
root@ubuntu:~# docker run ubuntu cat /sys/fs/cgroup/pids/pids.max
max
root@ubuntu:~# vim /etc/docker/daemon.json
root@ubuntu:~# cat /etc/docker/daemon.json
{
  "ipv6": true,
  "fixed-cidr-v6": "fd00::/80",
  "exec-opts": ["native.cgroupdriver=systemd"]
}
root@ubuntu:~# systemctl restart docker
root@ubuntu:~# docker run ubuntu cat /sys/fs/cgroup/pids/pids.max
7093

[rhatdan]

Which probably means that the issue is with systemd, On cgroup v1, so not likely to be fixed.

[jjenne-cisco]

Yes, thanks, you're right

root@ubuntu:~# podman run --pids-limit 0 ubuntu cat /sys/fs/cgroup/pids/pids.max
7093
root@ubuntu:~# echo 'DefaultTasksMax=infinity' >> /etc/systemd/system.conf
root@ubuntu:~# systemctl daemon-reexec
root@ubuntu:~# podman run --pids-limit 0 ubuntu cat /sys/fs/cgroup/pids/pids.max
max
root@ubuntu:~# podman run ubuntu cat /sys/fs/cgroup/pids/pids.max
2048

As you say, systemd assigns the pids/tasks max when it creates the container cgroup. Therefore in cgroups v1 we get "unlimited pids" only when podman is passed the "unlimited pids" argument and also systemd config is added to set "unlimited pids" by default.

I think what has been confusing is that while the help output for --pids-limit 0 says "for unlimited", in reality what it means is "if specified podman will not overwrite whatever value has been set by the cgroup driver".

[legaul-cisco]

Hey all :)

(disclaimer: I work with @jjenne-cisco and share the interest in this behaviour)

Some of my thoughts around the above (and referring to https://docs.podman.io/en/latest/markdown/options/pids-limit.html):

• Podman has a default pid limit
  • The docs say "The default is 4096 on systems that support “pids” cgroup controller."
  • Was the default limit changed from 2048 to 4096 at some point?
  • This is one of the few ways podman's behaviour deviates from docker, which has no default limit
  • I'm not sure why there's a default pid limit when there's no defaults for other limits such as memory - how is podman supposed to know what a sensible default is for all containers?
• The default pids limit can be removed by passing --pids-limit -1 or --pids-limit 0
  • The podman docs say "Set to -1 to have unlimited pids for the container."
  • I think --pids-limit 0 is allowed for podman backwards compatibility maybe?
  • Docker similarly supports --pids-limit with "Tune container pids limit (set -1 for unlimited)"
• Despite what the docs say, --pids-limit -1 isn't actually allowing unlimited pids in the container
  • When using the systemd cgroup driver there is a limit being set by systemd
  • systemd as the cgroup driver is the default with podman, but not docker on cgroups v1?
  • I'm guessing --pids-limit -1 is just being interpreted by podman as "don't set anything in pids.max", rather than "set max in pids.max"?
  • --pids-limit max is rejected by podman
• The behaviour on cgroups v2 seems to be different, i.e. no default limit being set by systemd (unsure why)

My vote/interpretation would be:

• Podman should have no default pid limit (systemd may set the value, as seen here)
• --pids-limit -1 (or perhaps --pids-limit max?) should explicitly set the limit to 'max' (overwriting whatever systemd might set)
• Question over whether this pid limit should be being set by systemd, although I suspect this is intended behaviour

As things stand, the best we can do is pass --pids-limit <N> where <N> is some arbitrarily large pid limit... It seems strange that this would be the recommended solution rather than an argument documented to set the limit to unlimited ('max').

Any thoughts welcome :)

[rhatdan]

Code says the default should be 2048
DefaultPidsLimit = 2048

[rhatdan]

On Fedora 36 with cgroups v2, I see:

$ podman run alpine cat /sys/fs/cgroup/pids.max
2048
$ podman run --pids-limit -1 alpine cat /sys/fs/cgroup/pids.max
max
$ podman run --pids-limit 0 alpine cat /sys/fs/cgroup/pids.max
max

[legaul-cisco]

Yeah, agreed this isn't a problem on cgroups v2 (not exactly sure why there's a difference). Podman still supports cgroups v1 though right?

[rhatdan]

Yes