CAdvisor with rootless docker does not discover subcontainers

[Mark90]

Hi,

I'm running into a problem when combining CAdvisor with rootless docker and hope it's ok to ask here. I'll explain in detail.

Context

I have an Azure VM running docker in rootless mode using rootlesskit.

Versions:

OS: Ubuntu 20.04.3 LTS  (kernel 5.15.0-1023-azure)
Docker: 20.10.12
Rootlesskit: 0.14.4

The non-root user running docker is called azureuser with uid/gid 1000/1000.

For reference these are the docker contexts of the non-root and root user:

azureuser@myvm:~$ docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT                     KUBERNETES ENDPOINT   ORCHESTRATOR
default *   Current DOCKER_HOST based configuration   unix:///run/user/1000/docker.sock                         swarm
rootless    Rootless mode                             unix:///run/user/1000/docker.sock

root@myvm:~# docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT   ORCHESTRATOR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                         swarm

My goal is to run a CAdvisor container as azureuser such that it can monitor other containers executing under azureuser.

This "almost" works: when browsing CAdvisor's UI it shows the number of containers, but there is no "subcontainer" section with actual information about the other containers .

This problem is limited to the rootless setup; when I run the same setup (CAdvisor and another dummy container) as root, then the UI does show information of other containers for root.

Setup

How I'm currently running this for the azureuser:

1. Dummy container sleep-container-azureuser

docker run --rm --detach --name sleep-container-azureuser alpine sh -c 'sleep 123123123'

2. CAdvisor container cadvisor-azureuser at port 9338

docker run \
	--volume=/:/rootfs:ro \
	--volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
	--volume=/run/user/1000/docker.sock:/var/run/docker.sock:ro \
	--volume=/home/azureuser/.local/share/docker/:/home/azureuser/.local/share/docker:ro \
	--device=/dev/kmsg \
	--privileged \
	--publish=9338:8080 \
	--name=cadvisor-azureuser \
	--rm \
	gcr.io/cadvisor/cadvisor:v0.47.1

And the equivalent setup for root:

1. Dummy container sleep-container-root

docker run --rm --detach --name sleep-container-root alpine sh -c 'sleep 123123123'

2. CAdvisor container cadvisor-root at port 9339

docker run \
	--volume=/:/rootfs:ro \
	--volume=/sys:/sys:ro \
	--volume=/var/run/docker.sock:/var/run/docker.sock:ro \
	--device=/dev/kmsg \
	--privileged \
	--publish=9339:8080 \
	--name=cadvisor-root \
	--rm \
	gcr.io/cadvisor/cadvisor:v0.47.1

Problem

I'll use CAdvisor's /metrics output to demonstrate the issue.

The cadvisor-azureuser (port 9338) container does not have information on the subcontainers;

azureuser@myvm:~$ curl -s localhost:9338/metrics | grep -c cadvisor-azureuser
0
azureuser@myvm:~$ curl -s localhost:9338/metrics | grep -c sleep-container
0

The cadvisor-root (port 9339) container does:

azureuser@myvm:~$ curl -s localhost:9339/metrics | grep -c cadvisor-root
65
azureuser@myvm:~$ curl -s localhost:9339/metrics | grep -c sleep-container
65

Or through the UI:

cadvisor-azureuser does show number of containers, but no subcontainers:

cadvisor-root does show subcontainers:

What I've tried

• Inspecting the docker socket from within the cadvisor instance does show container information:

azureuser@myvm:~$ docker exec cadvisor-azureuser sh -c 'apk update && apk add curl jq'
...
azureuser@myvm:~$ docker exec -i -t cadvisor-azureuser sh -c 'curl --silent --unix-socket /var/run/docker.sock http://localhost/containers/json | jq ".[] | .Names" | grep azureuser'
  "/sleep-container-azureuser"
  "/cadvisor-azureuser"

root@myvm:~# docker exec cadvisor-root sh -c 'apk update && apk add curl jq'
...
root@myvm:~# docker exec -i -t cadvisor-root sh -c 'curl --silent --unix-socket /var/run/docker.sock http://localhost/containers/json | jq ".[] | .Names" | grep
root'
  "/sleep-container-root"
  "/cadvisor-root"

• But this socket is probably not used by CAdvisor to gather subcontainer information. I started reading CAdvisor's source code to see how it does, and I think it's related to missing cgroup information. But I'm out of my depth here :)
• I've tried mounting all sorts of possibly relevant host folders into cadvisor-azureuser
• I've searched the internet with many different queries, it looks like this setup should "just work"

Perhaps someone reading along here has used a similar setup, or has an idea what the problem could be. If you made it to the end of this post, sorry for the wall of text.

Kind regards,
Mark

[jotespeh]

Hey,

I encountered the same issue with a similar setup (Docker Rootless 20+ and Ubuntu 20.04, but standalone cAdvisor).
All "meta" metrics were recognized, but no subcontainers reported. Also, when using docker stats all container reported the same metrics.

After upgrading to Ubuntu 22.04 the metrics for subcontainers are now properly reported.

My suspicion is that the underlying issue might have to do with cgroups v1 vs. v2. After Ubuntu 21.10 rootlesskit uses cgroups v2. See also: https://docs.docker.com/config/containers/runmetrics/#enumerate-cgroups
Unfortunately I did not dig deeper as simply dist upgrading helped in my case. Hope that it also works for you.

Best,

Jakob

[Mark90]

Hi jotespeh,

Thank you!! In Ubuntu 22.04 it indeed works as expected!

To confirm I created 2 new VMs, one with Ubuntu 20.04 and one with 22.04, then configured rootless docker on both. In the 20.04 VM CAdvisor does not show subcontainers, after a dist-upgrade to 22.04 it does.

VM with 22.04 installed -> subcontainers visible
Versions

OS: Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-1033-azure x86_64)
Docker: 23.0.1, build a5ee5b1
Rootlesskit: 1.1.0

VM with 20.04 installed -> subcontainers NOT visible
Versions

OS: Ubuntu 20.04.5 LTS (GNU/Linux 5.15.0-1033-azure x86_64)
Docker: 23.0.1, build a5ee5b1
Rootlesskit: 1.1.0

VM with 20.04 installed and upgraded to 22.04 -> subcontainers are now visible!

I too will not dig deeper as upgrading to Ubuntu 22.04 is an acceptable solution for me. Closing this.

[finkr]

After some investigation, the issue seems related to Control Groups v1 support.

Some people noticed that Ubuntu 22.04 works... which uses cgroups v2

Docker doesn't use cgroups delegation in rootless mode (see docker info | grep -i CGroup), probably because it's not safe, according to rootlesscontaine.rs.

cAdvisor doesn't work on RHEL 8 with cgroups v1.
cadvisor work on RHEL8 after switching to Cgroups v2 (https://access.redhat.com/solutions/6898151)

RHEL9 has cgroups v2 enabled by default.

As a workaround, we used Telegraf because we don't want to migrate our servers to cgroups v2