OCI runtime create failed - open /proc/sys/kernel/domainname: read-only file system: unknown

[jankeromnes]

Bug description

A friend of mine is trying to use Gitpod for a customer project using openldap and docker-compose. The Docker image being used is osixia/openldap:stable.

They're currently hitting this error when running docker-compose up:

ERROR: for openldap  Cannot start service openldap: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key kernel.domainname: open /proc/sys/kernel/domainname: read-only file system: unknown

Is this a potential bug in Gitpod's Docker-in-Docker setup?

Steps to reproduce

1. Have a docker-compose.yml similar to this one:

docker-compose.yml

version: "3.9"
services:
  openldap:
    image: osixia/openldap:stable
    container_name: openldap
    environment:
      LDAP_LOG_LEVEL: "256"
      LDAP_ORGANISATION: "Example Inc."
      LDAP_DOMAIN: "exampledomain"
      LDAP_BASE_DN: ""
      LDAP_ADMIN_PASSWORD: "admin"
      LDAP_CONFIG_PASSWORD: "config"
      LDAP_READONLY_USER: "false"
      #LDAP_READONLY_USER_USERNAME: "readonly"
      #LDAP_READONLY_USER_PASSWORD: "readonly"
      LDAP_RFC2307BIS_SCHEMA: "false"
      LDAP_BACKEND: "mdb"
      LDAP_TLS: "true"
      LDAP_TLS_CRT_FILENAME: "ldap.crt"
      LDAP_TLS_KEY_FILENAME: "ldap.key"
      LDAP_TLS_DH_PARAM_FILENAME: "dhparam.pem"
      LDAP_TLS_CA_CRT_FILENAME: "ca.crt"
      LDAP_TLS_ENFORCE: "false"
      LDAP_TLS_CIPHER_SUITE: "SECURE256:-VERS-SSL3.0"
      LDAP_TLS_VERIFY_CLIENT: "demand"
      LDAP_REPLICATION: "false"
      #LDAP_REPLICATION_CONFIG_SYNCPROV: 'binddn="cn=admin,cn=config" bindmethod=simple credentials="$$LDAP_CONFIG_PASSWORD" searchbase="cn=config" type=refreshAndPersist retry="60 +" timeout=1 starttls=critical'
      #LDAP_REPLICATION_DB_SYNCPROV: 'binddn="cn=admin,$$LDAP_BASE_DN" bindmethod=simple credentials="$$LDAP_ADMIN_PASSWORD" searchbase="$$LDAP_BASE_DN" type=refreshAndPersist interval=00:00:00:10 retry="60 +" timeout=1 starttls=critical'
      #LDAP_REPLICATION_HOSTS: "#PYTHON2BASH:['ldap://ldap.example.org','ldap://ldap2.example.org']"
      KEEP_EXISTING_CONFIG: "false"
      LDAP_REMOVE_CONFIG_AFTER_SETUP: "true"
      LDAP_SSL_HELPER_PREFIX: "ldap"
    tty: true
    stdin_open: true
    command: [--copy-service, --loglevel, debug] #   --loglevel, debug
    volumes:
      - /var/lib/ldap
      - /etc/ldap/slapd.d
      - /container/service/slapd/assets/certs/
      - ./deploy/small.schema:/container/service/slapd/assets/config/bootstrap/schema/small.schema
      - ./deploy/fake.ldif:/container/service/slapd/assets/config/bootstrap/ldif/data.ldif

2. Try running docker-compose up

Workspace affected

No response

Expected behavior

openldap should start without errors.

This should be possible in principle, because when you run it like so (without docker-compose), it seems to work fine:

docker run osixia/openldap:stable

Example repository

No response

Anything else?

Here is a probable cause for this issue.

[MatthewHallCom]

We're seeing this as well with docker-compose inside of Gitpod. It's very inconsistent

[jankeromnes]

@matthewhall-ca Is it the same error for you at the end of the error message? (I.e. write sysctl key kernel.domainname: open /proc/sys/kernel/domainname: read-only file system: unknown)

[MatthewHallCom]

Apologies, no, we're seeing
ERROR: for device-web Cannot start service device-web: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "proc" to rootfs at "/proc" caused: mount through procfd: operation not permitted: unknown

[kylos101]

Hi 👋 , I tried to recreate this issue using the above docker-compose.yml, and it works in both the US and EU cluster (I tried with many workspaces). I even ctrl + c and ran docker-compose up many times, but was unable to recreate the problem.

@jankeromnes is it possible for them to share a public repo from which they are able to reliably recreate the problem?

[jankeromnes]

Many thanks @kylos101 for investigating this issue, and even creating a minimal reproduction case! 🙏 💯

I believe their code is private, so I'm not sure how to effectively make progress here. I'll ask if they can see significant differences between their setup and https://github.com/kylos101/OCI-runtime-create-failed -- but otherwise, I don't see a clear path forward. Maybe a call with some live debugging over screenshare could work, but since this issue isn't currently a blocker for them, I'd be more inclined to pause the investigation for now and wait for more elements (e.g. maybe from someone else who can reproduce the same issue in the future).

[kylos101]

@jankeromnes the other thing I would be curious about is - does this only happen during peak times, or does it happen during non-peak times, too.

I plan to pause the investigation for now, until we get more information.

[konne]

next one, just now, workspace: vizlib-gucci-jgm70a009op