Creating a pod has a probability of failure

[wzshiming]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  namespace: default
spec:
  replicas: 10
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - args:
        - -c
        - sleep inf
        command:
        - /bin/sh
        image: busybox
        imagePullPolicy: IfNotPresent
        name: test
        ports:
        - containerPort: 4444
          name: test
          protocol: TCP
        resources:
          limits:
            cpu: 10m
            memory: 10Mi
          requests:
            cpu: 10m
            memory: 10Mi

This will not happen if remove the memory limit or change it to 100MI

It's the fault here

runc/libcontainer/cgroups/fscommon/fscommon.go

Lines 42 to 51 in dbbe7e6

	func retryingWriteFile(fd *os.File, data string) error {
	for {
	_, err := fd.Write([]byte(data))
	if errors.Is(err, unix.EINTR) {
	logrus.Infof("interrupted while writing %s to %s", data, fd.Name())
	continue
	}
	return err
	}
	}

Maybe it's missing a condition check? unix.EBUSY

func retryingWriteFile(filename string, data []byte, perm os.FileMode) error {
	for {
		err := ioutil.WriteFile(filename, data, perm)
		if errors.Is(err, unix.EINTR) || errors.Is(err, unix.EBUSY) {
			logrus.Infof("interrupted while writing %s to %s", string(data), filename)
			continue
		}
		return err
	}
}

May I ask if this is a known problem? if not, I will try to fix it

[cyphar]

It's possible that the kernel can give spurious EBUSYs when setting container limits, so we might need to handle this. The problem is that it's not always correct to ignore EBUSY and loop endlessly -- you can get EBUSY in certain contexts where you're trying to modify limits in a cgroup which already has processes inside it.

[kolyshkin]

Some kernels return EBUSY on memory limit being set too low (== lower than the current usage). My guess is with the increased footprint on memory, runc init no longer fits in 10M.

What we could do is check the current usage on EBUSY and report a less cryptic error in such a case. @wzshiming do you want to test this?

[kolyshkin]

@wzshiming what kernel are you using?

Can you reproduce this issue with bare runc (i.e. no containerd/kubernetes etc)? Please provide the detailed steps needed to reproduce this.

[wzshiming]

@kolyshkin

# cat container-config.json
{
  "metadata": {
      "name": "busybox"
  },
  "image": {
      "image": "busybox"
  },
  "command": [
      "/bin/sh"
  ],
  "args" : [
      "-c",
      "sleep inf"
  ],

  "log_path": "0.log",
  "linux": {
    "resources" : {
       "cpu_period": 100,
       "cpu_quota": 100,
       "cpu_shares": 100,
       "memory_limit_in_bytes": 1000,
       "oom_score_adj": 1000
    }
  }
}

# cat pod-config.json
{
    "metadata": {
        "name": "test-0",
        "namespace": "default"
    },
    "log_directory": "/tmp",
    "linux": {
    }
}

[kolyshkin]

It seems you are trying to start a container with memory limit of 1000 bytes. This obviously won't work, but it's not a bug. A bug might be a somewhat cryptic error message (as I suggested above in #2736 (comment)).

@wzshiming again, what kernel version are you using?

[wzshiming]

@kolyshkin
Linux 3.10.0-1160.11.1.el7.x86_64 x86_64

[kolyshkin]

Reproduced locally (with memory limit set too low).

On cgroupv1 system:

kir@ubu2004:~/git/runc/tst$ grep -B2 -A2 memory config.json 
	"linux": {
		"resources": {
			"memory": {
				"limit": 1000
			},
kir@ubu2004:~/git/runc/tst$ sudo ../runc run xe3
ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: failed to write "1000": write /sys/fs/cgroup/memory/user.slice/user-1000.slice/session-1011.scope/xe3/memory.limit_in_bytes: device or resource busy 

The funny thing is, this is the best runc does -- it actually shows some error which is somewhat relevant (at least it mentions memory limit).

We do even worse with fs cgroup driver:

kir@ubu2004:~/git/runc/tst$ sudo ../runc --systemd-cgroup run xe3
ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:348: copying bootstrap data to pipe caused: write init-p: broken pipe 
kir@ubu2004:~/git/runc/tst$ sudo ../runc --systemd-cgroup run xe3
ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:352: getting the final child's pid from pipe caused: EOF 

...or on cgroupv2, for that matter:

[kir@kir-rhat runc-tst]$ sudo ../runc/runc --systemd-cgroup run xx4
ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: read init-p: connection reset by peer 
[kir@kir-rhat runc-tst]$ sudo ../runc/runc --systemd-cgroup run xx4
ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: read init-p: connection reset by peer 
[kir@kir-rhat runc-tst]$ sudo ../runc/runc  run xx4
ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:484: writing syncT 'resume' caused: write init-p: broken pipe 
[kir@kir-rhat runc-tst]$ sudo ../runc/runc  run xx4
ERRO[0000] container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: read init-p: connection reset by peer 

Looking into a fix (to provide a sane error).

[wzshiming]

Thanks
In my case, it seems to only appear when more containers are created at once. If there is no error when creating one by one, so I think this is like a bug.

[kolyshkin]

@wzshiming if possible, can you test your setup with the patches from #2812 and provide us with errors runc gives you (it should be something like unable to set memory limit to xxx (current usage: yyy, peak usage: zzz))

[wzshiming]

@kolyshkin

Originally it was 10Mi memory, which can trigger this effect. Today I will try again and need 3Mi memory to trigger.

Is there a possibility that it is just above a certain threshold. There is no problem when creating a single pod, but when the number is large, it exceeds this critical value.

Checkout to runc compiled by your PR, doesn't seem to work.