Unable to change host-injections when bind-mounting CVMFS from a host that has non-default host-injections location

[casparvl]

In my build I'm getting a lot of

mktemp: failed to create file via template '/cvmfs/software.eessi.io/versions/2023.06/init/eessi_defaults.XXXXXX': Permission denied
/gpfs/work1/1/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/event_7c214fd0-720b-11f0-84af-0ecb387b6cc4/run_000/linux_x86_64_amd_zen4/eessi.io-2023.06-software/software-layer-scripts/install_scripts.sh: line 51: : No such file or directory
sed command failed
mktemp: failed to create file via template '/cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash.XXXXXX': Permission denied
/gpfs/work1/1/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/event_7c214fd0-720b-11f0-84af-0ecb387b6cc4/run_000/linux_x86_64_amd_zen4/eessi.io-2023.06-software/software-layer-scripts/install_scripts.sh: line 51: : No such file or directory
sed command failed
mktemp: failed to create file via template '/cvmfs/software.eessi.io/versions/2023.06/init/lmod/csh.XXXXXX': Permission denied
/gpfs/work1/1/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/event_7c214fd0-720b-11f0-84af-0ecb387b6cc4/run_000/linux_x86_64_amd_zen4/eessi.io-2023.06-software/software-layer-scripts/install_scripts.sh: line 51: : No such file or directory
sed command failed
mktemp: failed to create file via template '/cvmfs/software.eessi.io/versions/2023.06/init/lmod/fish.XXXXXX': Permission denied
/gpfs/work1/1/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/event_7c214fd0-720b-11f0-84af-0ecb387b6cc4/run_000/linux_x86_64_amd_zen4/eessi.io-2023.06-software/software-layer-scripts/install_scripts.sh: line 51: : No such file or directory
sed command failed
mktemp: failed to create file via template '/cvmfs/software.eessi.io/versions/2023.06/init/lmod/ksh.XXXXXX': Permission denied
/gpfs/work1/1/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/event_7c214fd0-720b-11f0-84af-0ecb387b6cc4/run_000/linux_x86_64_amd_zen4/eessi.io-2023.06-software/software-layer-scripts/install_scripts.sh: line 51: : No such file or directory
sed command failed
mktemp: failed to create file via template '/cvmfs/software.eessi.io/versions/2023.06/init/lmod/zsh.XXXXXX': Permission denied
/gpfs/work1/1/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/event_7c214fd0-720b-11f0-84af-0ecb387b6cc4/run_000/linux_x86_64_amd_zen4/eessi.io-2023.06-software/software-layer-scripts/install_scripts.sh: line 51: : No such file or directory
sed command failed
mktemp: failed to create file via template '/cvmfs/software.eessi.io/versions/2023.06/init/modules/EESSI/2023.06.lua.XXXXXX': Permission denied
/gpfs/work1/1/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/event_7c214fd0-720b-11f0-84af-0ecb387b6cc4/run_000/linux_x86_64_amd_zen4/eessi.io-2023.06-software/software-layer-scripts/install_scripts.sh: line 51: : No such file or directory
sed command failed
mktemp: failed to create file via template '/cvmfs/software.eessi.io/versions/2023.06/init/easybuild/eb_hooks.py.XXXXXX': Permission denied
/gpfs/work1/1/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/event_7c214fd0-720b-11f0-84af-0ecb387b6cc4/run_000/linux_x86_64_amd_zen4/eessi.io-2023.06-software/software-layer-scripts/install_scripts.sh: line 51: : No such file or directory
sed command failed

and later

ERROR: Failed to create directory /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/modules/all: [Errno 13] Permission denied: '/cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen4'

and even later

grep: /cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/driver_version.txt: Permission denied
ESC[33mThe host GPU driver libraries version have changed. Now its: (v575.57.08)ESC[0m
ESC[33mCleaning out outdated symlinks.ESC[0m
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/cuda_version.txt': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/driver_version.txt': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/host': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/latest': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libEGL.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libEGL.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libEGL_nvidia.so.0': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGL.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGL.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLESv1_CM.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLESv1_CM.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLESv1_CM_nvidia.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLESv2.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLESv2.so.2': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLESv2_nvidia.so.2': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLX.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLX.so.0': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLX_nvidia.so.0': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLdispatch.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libGLdispatch.so.0': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libOpenCL.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libOpenGL.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libOpenGL.so.0': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libcuda.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libcuda.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libcudadebugger.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvcuvid.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvcuvid.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-cfg.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-cfg.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-egl-wayland.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-eglcore.so.555.42.06': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-encode.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-encode.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-fbc.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-fbc.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-glcore.so.555.42.06': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-glsi.so.555.42.06': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-glvkspirv.so.555.42.06': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-gpucomp.so.555.42.06': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-gtk3.so.555.42.06': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-ml.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-ml.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-nvvm.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-nvvm.so.4': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-opencl.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-opticalflow.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-ptxjitcompiler.so': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-ptxjitcompiler.so.1': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-rtcore.so.555.42.06': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvidia-tls.so.555.42.06': Permission denied
rm: cannot remove '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host/libnvoptix.so.1': Permission denied
ESC[31mERROR: Unable to remove files under '/cvmfs/software.eessi.io/host_injections/nvidia/x86_64/host'.ESC[0m

and even later:

== FAILED: Installation ended unsuccessfully: Failed to chmod/chown several paths: ['/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/software/CUDA/12.1.1',

This is on a host that has CVMFS on the host, so that https://github.com/EESSI/software-layer-scripts/pull/50/files causes it to be bind-mounted. The host-injections dir on the host is in a non-default location (i.e. NOT /opt/eessi). But note the last failure, which is outside of host-injections actually.

The singularity command from the logs is:

singularity  run --nv --contain --fusemount container:unionfs -o cow /tmp/software.eessi.io/overlay-upper=RW:/cvmfs_ro/software.eessi.io=RO /cvmfs/software.eessi.io /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb5V/eessi.09ZtPJ2M1s/ghcr.io_eessi_build_node_debian12.sif /gpfs/work1/1/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/event_7c214fd0-720b-11f0-84af-0ecb387b6cc4/run_000/linux_x86_64_amd_zen4/eessi.io-2023.06-software/software-layer-scripts/install_software_layer.sh --build-logs-dir /projects/eessibot/eessi-bot-surf/buildlogs --shared-fs-path /projects/eessibot/eessi-bot-surf/SHARED

[casparvl]

It's hard to mimic the exact environment of the build container, but...

mkdir -p /tmp/var-lib-cvmfs
mkdir -p /tmp/var-run-cvmfs
mkdir -p /tmp/software.eessi.io/overlay-upper
mkdir -p /tmp/singularity-tmp/
export SINGULARITY_TMPDIR=/tmp/singularity-tmp/
SINGULARITY_BIND=/usr/hpc,/projects/eessibot/eessi-bot-surf/buildlogs,/projects/eessibot/eessi-bot-surf/SHARED,/cvmfs/software.eessi.io:/cvmfs_ro/software.eessi.io,/tmp/var-lib-cvmfs:/var/lib/cvmfs,/tmp/var-run-cvmfs:/var/run/cvmfs
singularity --verbose run --bind /cvmfs/software.eessi.io:/cvmfs_ro/software.eessi.io --no-mount tmp --nv --contain --fusemount "container:unionfs -o cow /tmp/software.eessi.io/overlay-upper=RW:/cvmfs_ro/software.eessi.io=RO /cvmfs/software.eessi.io" /tmp/singularity/build-node_debian12.sif

Works. I.e. with this, I can

Apptainer> touch /cvmfs/software.eessi.io/foo
Apptainer> exit
exit
[eessibot@tcn470 ~]$ ls -al /tmp/software.eessi.io/overlay-upper/
total 0
drwxr-x---. 2 eessibot eessibot 80 Aug  6 09:50 .
drwxr-x---. 3 eessibot eessibot 60 Aug  6 09:25 ..
-rw-r-----. 1 eessibot eessibot  0 Aug  6 09:50 foo

So... it should be possible to get this to work. Do note that I had to pass --bind /cvmfs/software.eessi.io:/cvmfs_ro/software.eessi.io explicitely on the command line, having it in SINGULARITY_BIND only would result in

Failed to open /cvmfs_ro/software.eessi.io/: No such file or directory. Aborting!

When trying to start the container (maybe it tries the unionfs first, before the dir is mounted? Whereas with the explicit --bind option, it may be bound earlier?)

[casparvl]

Hmmm this also works:

$ ./eessi_container.sh -r software.eessi.io,mount=bind --access rw
Using /scratch-local/eessibot.13590511/eessi.k7qnVSfjbC as tmp directory (to resume session add '--resume /scratch-local/eessibot.13590511/eessi.k7qnVSfjbC').
Pulling container image from docker://ghcr.io/eessi/build-node:debian12 to /scratch-local/eessibot.13590511/eessi.k7qnVSfjbC/ghcr.io_eessi_build_node_debian12.sif
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
2025/08/06 10:02:59  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:02:59  warn xattr{/tmp/singularity-tmp/build-temp-3209213530/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:03:01  warn xattr{var/lib/apt/lists/auxfiles} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:03:01  warn xattr{/tmp/singularity-tmp/build-temp-3209213530/rootfs/var/lib/apt/lists/auxfiles} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:03:02  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:03:02  warn xattr{/tmp/singularity-tmp/build-temp-3209213530/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:03:10  warn xattr{var/log/apt/term.log} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:03:10  warn xattr{/tmp/singularity-tmp/build-temp-3209213530/rootfs/var/log/apt/term.log} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:03:10  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:03:10  warn xattr{/tmp/singularity-tmp/build-temp-3209213530/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:03:12  warn xattr{var/log/apt/term.log} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:03:12  warn xattr{/tmp/singularity-tmp/build-temp-3209213530/rootfs/var/log/apt/term.log} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:03:12  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:03:12  warn xattr{/tmp/singularity-tmp/build-temp-3209213530/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
[===================================================================================================================================================================================================================================================================================] 100 % 0s
TMPDIR is not empty (/scratch-local/eessibot.13590511) and TMP_IN_CONTAINER (/tmp) is not a prefix of TMPDIR: adding bind mount for TMPDIR
Launching container with command (next line):
singularity -q shell  --fusemount container:unionfs -o cow /tmp/software.eessi.io/overlay-upper=RW:/cvmfs_ro/software.eessi.io=RO /cvmfs/software.eessi.io /scratch-local/eessibot.13590511/eessi.k7qnVSfjbC/ghcr.io_eessi_build_node_debian12.sif
Ignoring invalid max threads value 4294967295 > max (100000).
(failed reverse-i-search)`touch': ^C
Apptainer> touch /cvmfs/software.eessi.io/foo
Apptainer> exit
$ ls /scratch-local/eessibot.13590511/eessi.k7qnVSfjbC/software.eessi.io/overlay-upper/
foo

So... why am I having trouble if this is done in a build pipeline?

[casparvl]

Even the version from the merge commit of #52 works:

$ ./eessi_container.sh -r software.eessi.io --access rw
Using /scratch-local/eessibot.13590511/eessi.aa4kY6NolL as tmp directory (to resume session add '--resume /scratch-local/eessibot.13590511/eessi.aa4kY6NolL').
Pulling container image from docker://ghcr.io/eessi/build-node:debian12 to /scratch-local/eessibot.13590511/eessi.aa4kY6NolL/ghcr.io_eessi_build_node_debian12.sif
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
2025/08/06 10:10:43  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:10:43  warn xattr{/tmp/singularity-tmp/build-temp-1666671750/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:10:45  warn xattr{var/lib/apt/lists/auxfiles} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:10:45  warn xattr{/tmp/singularity-tmp/build-temp-1666671750/rootfs/var/lib/apt/lists/auxfiles} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:10:45  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:10:45  warn xattr{/tmp/singularity-tmp/build-temp-1666671750/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:10:54  warn xattr{var/log/apt/term.log} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:10:54  warn xattr{/tmp/singularity-tmp/build-temp-1666671750/rootfs/var/log/apt/term.log} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:10:54  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:10:54  warn xattr{/tmp/singularity-tmp/build-temp-1666671750/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:10:56  warn xattr{var/log/apt/term.log} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:10:56  warn xattr{/tmp/singularity-tmp/build-temp-1666671750/rootfs/var/log/apt/term.log} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:10:56  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:10:56  warn xattr{/tmp/singularity-tmp/build-temp-1666671750/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
[===================================================================================================================================================================================================================================================================================] 100 % 0s
TMPDIR is not empty (/scratch-local/eessibot.13590511) and TMP_IN_CONTAINER (/tmp) is not a prefix of TMPDIR: adding bind mount for TMPDIR
Launching container with command (next line):
singularity -q shell  --fusemount container:unionfs -o cow /tmp/software.eessi.io/overlay-upper=RW:/cvmfs_ro/software.eessi.io=RO /cvmfs/software.eessi.io /scratch-local/eessibot.13590511/eessi.aa4kY6NolL/ghcr.io_eessi_build_node_debian12.sif
Ignoring invalid max threads value 4294967295 > max (100000).
Apptainer> touch /cvmfs/software.eessi.io/foo
Apptainer> exit
exit
$ ls -al /scratch-local/eessibot.13590511/eessi.aa4kY6NolL/software.eessi.io/overlay-upper/
total 1
drwxr-x---. 2 eessibot eessibot 4096 Aug  6 10:11 .
drwxr-x---. 4 eessibot eessibot 4096 Aug  6 10:11 ..
-rw-r-----. 1 eessibot eessibot    0 Aug  6 10:11 foo

I'm really puzzled at the errors we've seen in the build pipeline now.

[casparvl]

Note that touching host_injections does fail. But that's expected I guess:

Apptainer> ls -al /cvmfs/software.eessi.io/host_injections
lrwxrwxrwx. 1 nobody nogroup 25 Oct  3  2023 /cvmfs/software.eessi.io/host_injections -> /gpfs/admin/hpc/sw/EESSI/
Apptainer> ls -al /cvmfs/software.eessi.io/host_injections/
total 2
drwxr-xr-x. 4 nobody nogroup 4096 Mar 29  2024 .
drwxrwxr-x. 9 nobody nogroup 4096 Dec 22  2023 ..
drwxr-xr-x. 5 nobody nogroup 4096 Mar 29  2024 2023.06
drwxr-xr-x. 3 nobody nogroup 4096 Dec 22  2023 nvidia
Apptainer> touch /cvmfs/software.eessi.io/host_injections/foo
touch: cannot touch '/cvmfs/software.eessi.io/host_injections/foo': Permission denied

[casparvl]

Interesting... I'm still at 772e408. There, I can touch a file if I start an interactive eessi_container.sh:

export APPTAINER_CONFIG_FILE=/usr/hpc/etc/eessibot_apptainer.conf
export EESSI_REPOS_CFG_DIR_OVERRIDE=/projects/eessibot/eessi-bot-surf/jobs/2025.08/pr_1147/13575145/cfg
$ ./eessi_container.sh --verbose --access rw --container docker://ghcr.io/eessi/build-node:debian12 --repository eessi.io-2023.06-software --extra-bind-paths /dev --pass-through --contain --storage /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12 --nvidi
a all --host-injections /projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp
checking for duplicates: 'eessi.io-2023.06-software' and 'eessi.io-2023.06-software'
repo 'eessi.io-2023.06-software' is not an EESSI CVMFS repository...
Using /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn as tmp directory (to resume session add '--resume /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn').
EESSI_TMPDIR=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn
HOST_INJECTIONS=/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp
SINGULARITY_CACHEDIR=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/singularity_cache
Pulling container image from docker://ghcr.io/eessi/build-node:debian12 to /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/ghcr.io_eessi_build_node_debian12.sif
INFO:    Environment variable SINGULARITY_CACHEDIR is set, but APPTAINER_CACHEDIR is preferred
INFO:    Converting OCI blobs to SIF format
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Fetching OCI image...
111.0MiB / 111.0MiB [====================================================================================================================================================================================================================================================] 100 % 59.2 MiB/s 0s
14.5MiB / 14.5MiB [======================================================================================================================================================================================================================================================] 100 % 59.2 MiB/s 0s
50.5MiB / 50.5MiB [======================================================================================================================================================================================================================================================] 100 % 59.2 MiB/s 0s
38.7MiB / 38.7MiB [======================================================================================================================================================================================================================================================] 100 % 59.2 MiB/s 0s
121.7MiB / 121.7MiB [====================================================================================================================================================================================================================================================] 100 % 59.2 MiB/s 0s
46.2MiB / 46.2MiB [======================================================================================================================================================================================================================================================] 100 % 59.2 MiB/s 0s
34.3MiB / 34.3MiB [======================================================================================================================================================================================================================================================] 100 % 59.2 MiB/s 0s
7.8MiB / 7.8MiB [========================================================================================================================================================================================================================================================] 100 % 59.2 MiB/s 0s
INFO:    Extracting OCI image...
2025/08/06 10:32:26  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:32:26  warn xattr{/tmp/build-temp-2382207149/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:32:28  warn xattr{var/lib/apt/lists/auxfiles} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:32:28  warn xattr{/tmp/build-temp-2382207149/rootfs/var/lib/apt/lists/auxfiles} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:32:28  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:32:28  warn xattr{/tmp/build-temp-2382207149/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:32:36  warn xattr{var/log/apt/term.log} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:32:36  warn xattr{/tmp/build-temp-2382207149/rootfs/var/log/apt/term.log} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:32:36  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:32:36  warn xattr{/tmp/build-temp-2382207149/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:32:39  warn xattr{var/log/apt/term.log} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:32:39  warn xattr{/tmp/build-temp-2382207149/rootfs/var/log/apt/term.log} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:32:39  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:32:39  warn xattr{/tmp/build-temp-2382207149/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
INFO:    Inserting Apptainer configuration...
INFO:    Creating SIF file...
[===================================================================================================================================================================================================================================================================================] 100 % 0s
CONTAINER=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/ghcr.io_eessi_build_node_debian12.sif
EESSI_CVMFS_VAR_LIB=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-lib-cvmfs
EESSI_CVMFS_VAR_RUN=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-run-cvmfs
SINGULARITY_HOME=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/home:/home/eessibot
TMPDIR is not empty (/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12) and TMP_IN_CONTAINER (/tmp) is not a prefix of TMPDIR: adding bind mount for TMPDIR
BIND_PATHS=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_jo
b.7ab9ZoLb12/eessi.PpUxUFrlnn:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev
ADDITIONAL_CONTAINER_OPTIONS=--nv
BIND_PATHS=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_jo
b.7ab9ZoLb12/eessi.PpUxUFrlnn:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-log:/var/log,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/usr-local-cuda:/usr/local/cuda

BIND_PATHS before processing REPOSITORIES
  BIND_PATHS=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_
job.7ab9ZoLb12/eessi.PpUxUFrlnn:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-log:/var/log,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/usr-local-cuda:/usr/local/cuda

process CVMFS repo spec 'eessi.io-2023.06-software'
default.local --> /etc/cvmfs/default.local
eessi.io/eessi.io.pub --> /etc/cvmfs/keys/eessi.io/eessi.io.pub
eessi.io.conf --> /etc/cvmfs/domain.d/eessi.io.conf
BIND_PATHS after processing 'eessi.io-2023.06-software'
  BIND_PATHS=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_
job.7ab9ZoLb12/eessi.PpUxUFrlnn:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-log:/var/log,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/usr-local-cuda:/usr/local/cuda,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb1
2/eessi.PpUxUFrlnn/repos_cfg/default.local:/etc/cvmfs/default.local,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/repos_cfg/eessi.io/eessi.io.pub:/etc/cvmfs/keys/eessi.io/eessi.io.pub,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/repos_cfg/eessi.io.conf:/etc/cvm
fs/domain.d/eessi.io.conf

add fusemount options for CVMFS repo 'eessi.io-2023.06-software'
repo 'eessi.io-2023.06-software' is not an EESSI CVMFS repository...
bob eessi.io-2023.06-software
software.eessi.io
Using a bind mount for /cvmfs/software.eessi.io
TMP directory contents:
total 419864
-rwxr-x---. 1 eessibot eessibot 429940736 Aug  6 10:32 ghcr.io_eessi_build_node_debian12.sif
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:32 home
drwxr-x---. 3 eessibot eessibot       120 Aug  6 10:32 repos_cfg
drwxr-x---. 3 eessibot eessibot        60 Aug  6 10:32 singularity_cache
drwxr-x---. 4 eessibot eessibot        80 Aug  6 10:32 software.eessi.io
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:32 usr-local-cuda
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:32 var-lib-cvmfs
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:32 var-log
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:32 var-run-cvmfs
SINGULARITY_BIND=/cvmfs/software.eessi.io:/cvmfs_ro/software.eessi.io,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/h
ost-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/var-log:/var/log,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/usr-local-cud
a:/usr/local/cuda,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/repos_cfg/default.local:/etc/cvmfs/default.local,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/repos_cfg/eessi.io/eessi.io.pub:/etc/cvmfs/keys/eessi.io/eessi.io.pub,/tmp/eessibot/EESSI/eessi_job.7ab
9ZoLb12/eessi.PpUxUFrlnn/repos_cfg/eessi.io.conf:/etc/cvmfs/domain.d/eessi.io.conf,/cvmfs/cvmfs-config.cern.ch
Launching container with command (next line):
singularity  shell --nv --contain --fusemount container:unionfs -o cow /tmp/software.eessi.io/overlay-upper=RW:/cvmfs_ro/software.eessi.io=RO /cvmfs/software.eessi.io /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.PpUxUFrlnn/ghcr.io_eessi_build_node_debian12.sif
INFO:    Environment variable SINGULARITY_BIND is set, but APPTAINER_BIND is preferred
INFO:    Environment variable SINGULARITY_HOME is set, but APPTAINER_HOME is preferred
Ignoring invalid max threads value 4294967295 > max (100000).
Apptainer> touch /cvmfs/software.eessi.io/foo
Apptainer>

If, however, I do this with the --mode run argument and then -- touch /cvmfs/software.eessi.io/foo:

$ ./eessi_container.sh --verbose --access rw --mode run --container docker://ghcr.io/eessi/build-node:debian12 --repository eessi.io-2023.06-software --extra-bind-paths /dev --pass-through --contain --storage /tmp/eessibot/EESSI/eessi_job.7ab9ZoL
b12 --nvidia all --host-injections /projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp -- touch /cvmfs/software.eessi.io/foo
checking for duplicates: 'eessi.io-2023.06-software' and 'eessi.io-2023.06-software'
repo 'eessi.io-2023.06-software' is not an EESSI CVMFS repository...
Using /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp as tmp directory (to resume session add '--resume /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp').
EESSI_TMPDIR=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp
HOST_INJECTIONS=/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp
SINGULARITY_CACHEDIR=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/singularity_cache
Pulling container image from docker://ghcr.io/eessi/build-node:debian12 to /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/ghcr.io_eessi_build_node_debian12.sif
INFO:    Environment variable SINGULARITY_CACHEDIR is set, but APPTAINER_CACHEDIR is preferred
INFO:    Converting OCI blobs to SIF format
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Fetching OCI image...
50.5MiB / 50.5MiB [======================================================================================================================================================================================================================================================] 100 % 41.4 MiB/s 0s
121.7MiB / 121.7MiB [====================================================================================================================================================================================================================================================] 100 % 41.4 MiB/s 0s
14.5MiB / 14.5MiB [======================================================================================================================================================================================================================================================] 100 % 41.4 MiB/s 0s
111.0MiB / 111.0MiB [====================================================================================================================================================================================================================================================] 100 % 41.4 MiB/s 0s
34.3MiB / 34.3MiB [======================================================================================================================================================================================================================================================] 100 % 41.4 MiB/s 0s
7.8MiB / 7.8MiB [========================================================================================================================================================================================================================================================] 100 % 41.4 MiB/s 0s
46.2MiB / 46.2MiB [======================================================================================================================================================================================================================================================] 100 % 41.4 MiB/s 0s
38.7MiB / 38.7MiB [======================================================================================================================================================================================================================================================] 100 % 41.4 MiB/s 0s
INFO:    Extracting OCI image...
2025/08/06 10:33:45  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:33:45  warn xattr{/tmp/build-temp-518620842/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:33:47  warn xattr{var/lib/apt/lists/auxfiles} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:33:47  warn xattr{/tmp/build-temp-518620842/rootfs/var/lib/apt/lists/auxfiles} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:33:48  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:33:48  warn xattr{/tmp/build-temp-518620842/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:33:56  warn xattr{var/log/apt/term.log} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:33:56  warn xattr{/tmp/build-temp-518620842/rootfs/var/log/apt/term.log} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:33:56  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:33:56  warn xattr{/tmp/build-temp-518620842/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:33:58  warn xattr{var/log/apt/term.log} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:33:58  warn xattr{/tmp/build-temp-518620842/rootfs/var/log/apt/term.log} destination filesystem does not support xattrs, further warnings will be suppressed
2025/08/06 10:33:58  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/08/06 10:33:58  warn xattr{/tmp/build-temp-518620842/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
INFO:    Inserting Apptainer configuration...
INFO:    Creating SIF file...
[===================================================================================================================================================================================================================================================================================] 100 % 0s
CONTAINER=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/ghcr.io_eessi_build_node_debian12.sif
EESSI_CVMFS_VAR_LIB=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-lib-cvmfs
EESSI_CVMFS_VAR_RUN=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-run-cvmfs
SINGULARITY_HOME=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/home:/home/eessibot
TMPDIR is not empty (/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12) and TMP_IN_CONTAINER (/tmp) is not a prefix of TMPDIR: adding bind mount for TMPDIR
BIND_PATHS=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_jo
b.7ab9ZoLb12/eessi.x1yWaegcxp:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev
ADDITIONAL_CONTAINER_OPTIONS=--nv
BIND_PATHS=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_jo
b.7ab9ZoLb12/eessi.x1yWaegcxp:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-log:/var/log,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/usr-local-cuda:/usr/local/cuda

BIND_PATHS before processing REPOSITORIES
  BIND_PATHS=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_
job.7ab9ZoLb12/eessi.x1yWaegcxp:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-log:/var/log,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/usr-local-cuda:/usr/local/cuda

process CVMFS repo spec 'eessi.io-2023.06-software'
default.local --> /etc/cvmfs/default.local
eessi.io/eessi.io.pub --> /etc/cvmfs/keys/eessi.io/eessi.io.pub
eessi.io.conf --> /etc/cvmfs/domain.d/eessi.io.conf
BIND_PATHS after processing 'eessi.io-2023.06-software'
  BIND_PATHS=/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_
job.7ab9ZoLb12/eessi.x1yWaegcxp:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-log:/var/log,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/usr-local-cuda:/usr/local/cuda,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb1
2/eessi.x1yWaegcxp/repos_cfg/default.local:/etc/cvmfs/default.local,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/repos_cfg/eessi.io/eessi.io.pub:/etc/cvmfs/keys/eessi.io/eessi.io.pub,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/repos_cfg/eessi.io.conf:/etc/cvm
fs/domain.d/eessi.io.conf

add fusemount options for CVMFS repo 'eessi.io-2023.06-software'
repo 'eessi.io-2023.06-software' is not an EESSI CVMFS repository...
bob eessi.io-2023.06-software
software.eessi.io
Using a bind mount for /cvmfs/software.eessi.io
TMP directory contents:
total 419864
-rwxr-x---. 1 eessibot eessibot 429940736 Aug  6 10:34 ghcr.io_eessi_build_node_debian12.sif
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:34 home
drwxr-x---. 3 eessibot eessibot       120 Aug  6 10:34 repos_cfg
drwxr-x---. 3 eessibot eessibot        60 Aug  6 10:33 singularity_cache
drwxr-x---. 4 eessibot eessibot        80 Aug  6 10:34 software.eessi.io
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:34 usr-local-cuda
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:34 var-lib-cvmfs
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:34 var-log
drwxr-x---. 2 eessibot eessibot        40 Aug  6 10:34 var-run-cvmfs
SINGULARITY_BIND=/cvmfs/software.eessi.io:/cvmfs_ro/software.eessi.io,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-lib-cvmfs:/var/lib/cvmfs,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-run-cvmfs:/var/run/cvmfs,/projects/eessibot/eessi-bot-surf/SHARED/h
ost-injections-temp:/opt/eessi,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp:/tmp,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12,/dev,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/var-log:/var/log,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/usr-local-cud
a:/usr/local/cuda,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/repos_cfg/default.local:/etc/cvmfs/default.local,/tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/repos_cfg/eessi.io/eessi.io.pub:/etc/cvmfs/keys/eessi.io/eessi.io.pub,/tmp/eessibot/EESSI/eessi_job.7ab
9ZoLb12/eessi.x1yWaegcxp/repos_cfg/eessi.io.conf:/etc/cvmfs/domain.d/eessi.io.conf,/cvmfs/cvmfs-config.cern.ch
Launching container with command (next line):
singularity  run --nv --contain --fusemount container:unionfs -o cow /tmp/software.eessi.io/overlay-upper=RW:/cvmfs_ro/software.eessi.io=RO /cvmfs/software.eessi.io /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12/eessi.x1yWaegcxp/ghcr.io_eessi_build_node_debian12.sif touch /cvmfs/software.ees
si.io/foo
INFO:    Environment variable SINGULARITY_BIND is set, but APPTAINER_BIND is preferred
INFO:    Environment variable SINGULARITY_HOME is set, but APPTAINER_HOME is preferred
Ignoring invalid max threads value 4294967295 > max (100000).
/usr/bin/touch: cannot touch '/cvmfs/software.eessi.io/foo': Permission denied

I can reproduce the original issue!

Why these would be different, I have no idea...

[casparvl]

To see if we did anything different in eessi_container.sh when --mode run is passed, I tried instead to get an interactive shell through:

./eessi_container.sh --verbose --access rw --mode run --container docker://ghcr.io/eessi/build-node:debian12 --repository eessi.io-2023.06-software --extra-bind-paths /dev --pass-through --contain --storage /tmp/eessibot/EESSI/eessi_job.7ab9ZoL
b12 --nvidia all --host-injections /projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp -- /bin/bash
Apptainer> touch /cvmfs/software.eessi.io/foo
Apptainer>

So... then it works?!

[casparvl]

Ok, I've tried various different things, including trying to figure out if the overlay was mounted correctly in both cases. Running:

./eessi_container.sh --verbose --access rw --mode run --container docker://ghcr.io/eessi/build-node:debian12 --repository eessi.io-2023.06-software,mode=bind --extra-bind-paths /dev --pass-through --contain --storage /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12 --nvidia all --host-injections /projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp -- findmnt

or

./eessi_container.sh --verbose --access rw --mode run --container docker://ghcr.io/eessi/build-node:debian12 --repository eessi.io-2023.06-software,mode=bind --extra-bind-paths /dev --pass-through --contain --storage /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12 --nvidia all --host-injections /projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp -- /bin/bash
Apptainer > findmnt

both result in

| `-/cvmfs/software.eessi.io                                    cvmfs2                                                                                    fuse       ro,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
|   `-/cvmfs/software.eessi.io                                  fuse                                                                                      fuse       rw,nosuid,nodev,relatime,user_id=72648,group_id=72109

Which looks absolutely fine (and is the same between both approaches).

Launching with ... -- strace touch /cvmfs/software.eessi.io/foo I see:

openat(AT_FDCWD, "/cvmfs/software.eessi.io/foo", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666CernVM-FS: loading Fuse module... done
Ignoring invalid max threads value 4294967295 > max (100000).
) = -1 EACCES (Permission denied)
utimensat(AT_FDCWD, "/cvmfs/software.eessi.io/foo", NULL, 0) = -1 ENOENT (No such file or directory)
write(2, "touch: ", 7touch: )                  = 7
write(2, "cannot touch '/cvmfs/software.ee"..., 43cannot touch '/cvmfs/software.eessi.io/foo') = 43
write(2, ": Permission denied", 19: Permission denied)     = 19

One thing that stands out here is that the fuse module is loaded the first time /cvmfs/software.eessi.io is accessed. This made me think: what if I try to access it first, before trying a write? Maybe the fuse module is loaded immediately for an interactive shell, before I execute my first command. And indeed, first doing an ls before the touch resolves the issue:

$ cat wrap.sh
#!/bin/bash
ls -al /cvmfs/software.eessi.io/
touch /cvmfs/software.eessi.io/foo
ls -al /cvfms/software.eessi.io
ls -al /tmp/software.eessi.io/overlay-upper/
$ ./eessi_container.sh --verbose --access rw --mode run --container docker://ghcr.io/eessi/build-node:debian12 --repository eessi.io-2023.06-software,mode=bind --extra-bind-paths $PWD,/dev --pass-through --contain --storage /tmp/eessibot/EESSI/ee
ssi_job.7ab9ZoLb12 --nvidia all --host-injections /projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp -- $PWD/wrap.sh
...
total 4
drwxr-x---. 1 eessibot eessibot   40 Aug  6 11:51 .
drwxr-xr-x. 4 nobody   nogroup     0 Aug  6 10:29 ..
-rw-r--r--. 1 eessibot eessibot 1070 May 19 14:06 .cvmfsdirtab
-rw-r--r--. 1 eessibot eessibot  565 Dec  2  2023 README.eessi
lrwxrwxrwx. 1 eessibot eessibot   10 Oct  3  2023 host_injections -> /opt/eessi
drwxr-xr-x. 1 eessibot eessibot   21 Sep  5  2024 init
drwxr-xr-x. 1 eessibot eessibot   21 Jul 22  2024 versions
total 4
drwxr-x---. 1 eessibot eessibot   60 Aug  6 11:55 .
drwxr-xr-x. 4 nobody   nogroup     0 Aug  6 10:29 ..
-rw-r--r--. 1 eessibot eessibot 1070 May 19 14:06 .cvmfsdirtab
-rw-r--r--. 1 eessibot eessibot  565 Dec  2  2023 README.eessi
-rw-r-----. 1 eessibot eessibot    0 Aug  6 11:55 foo
lrwxrwxrwx. 1 eessibot eessibot   10 Oct  3  2023 host_injections -> /opt/eessi
drwxr-xr-x. 1 eessibot eessibot   21 Sep  5  2024 init
drwxr-xr-x. 1 eessibot eessibot   21 Jul 22  2024 versions
total 0
drwxr-x---. 2 eessibot eessibot  60 Aug  6 11:51 .
drwxr-x---. 4 eessibot eessibot 100 Aug  6 11:51 ..
-rw-r-----. 1 eessibot eessibot   0 Aug  6 11:51 foo

Note that this used the mode=bind for the repository, so we are talking to the host CVMFS here! (or should be, at least, I have to think of a way to confirm).

[casparvl]

Ah, some clarity. Now I have:

$ cat wrap.sh
#!/bin/bash
echo "Trying the root CVFMS repo"
ls -al /cvmfs/software.eessi.io/
touch /cvmfs/software.eessi.io/foo
ls -al /cvmfs/software.eessi.io
ls -al /tmp/software.eessi.io/overlay-upper/

echo "Trying a nested dir"
ls -al /cvmfs/software.eessi.io/versions
touch /cvmfs/software.eessi.io/versions/foo
ls -al /cvmfs/software.eessi.io/versions
ls -al /tmp/software.eessi.io/overlay-upper/

Then I run:

 ./eessi_container.sh --verbose --access rw --mode run --container docker://ghcr.io/eessi/build-node:debian12 --repository eessi.io-2023.06-software,mount=bind --extra-bind-paths $PWD,/dev --pass-through --contain --storage /tmp/eessibot/EESSI/eessi_job.7ab9ZoLb12 --nvidia all --host-injections /projects/eessibot/eessi-bot-surf/SHARED/host-injections-temp -- $PWD/wrap.sh

And I get:

Trying the root CVFMS repo
total 4
drwxr-x---. 1 eessibot eessibot   40 Aug  6 13:59 .
drwxr-xr-x. 4 nobody   nogroup     0 Aug  6 13:53 ..
-rw-r--r--. 1 nobody   nogroup  1070 May 19 14:06 .cvmfsdirtab
-rw-r--r--. 1 nobody   nogroup   565 Dec  2  2023 README.eessi
lrwxrwxrwx. 1 nobody   nogroup    25 Oct  3  2023 host_injections -> /gpfs/admin/hpc/sw/EESSI/
drwxr-xr-x. 1 nobody   nogroup    21 Sep  5  2024 init
drwxr-xr-x. 1 nobody   nogroup    21 Jul 22  2024 versions
total 4
drwxr-x---. 1 eessibot eessibot   60 Aug  6 13:59 .
drwxr-xr-x. 4 nobody   nogroup     0 Aug  6 13:53 ..
-rw-r--r--. 1 nobody   nogroup  1070 May 19 14:06 .cvmfsdirtab
-rw-r--r--. 1 nobody   nogroup   565 Dec  2  2023 README.eessi
-rw-r-----. 1 eessibot eessibot    0 Aug  6 13:59 foo
lrwxrwxrwx. 1 nobody   nogroup    25 Oct  3  2023 host_injections -> /gpfs/admin/hpc/sw/EESSI/
drwxr-xr-x. 1 nobody   nogroup    21 Sep  5  2024 init
drwxr-xr-x. 1 nobody   nogroup    21 Jul 22  2024 versions
total 0
drwxr-x---. 2 eessibot eessibot  60 Aug  6 13:59 .
drwxr-x---. 4 eessibot eessibot 100 Aug  6 13:59 ..
-rw-r-----. 1 eessibot eessibot   0 Aug  6 13:59 foo
Trying a nested dir
total 2
drwxr-xr-x. 1 nobody   nogroup  21 Jul 22  2024 .
drwxr-x---. 1 eessibot eessibot 60 Aug  6 13:59 ..
drwxr-xr-x. 1 nobody   nogroup  22 Oct  3  2023 2023.06
drwxr-xr-x. 1 nobody   nogroup  22 Jun 13 18:02 2025.06
touch: cannot touch '/cvmfs/software.eessi.io/versions/foo': Permission denied
total 2
drwxr-xr-x. 1 nobody   nogroup  21 Jul 22  2024 .
drwxr-x---. 1 eessibot eessibot 60 Aug  6 13:59 ..
drwxr-xr-x. 1 nobody   nogroup  22 Oct  3  2023 2023.06
drwxr-xr-x. 1 nobody   nogroup  22 Jun 13 18:02 2025.06
total 0
drwxr-x---. 2 eessibot eessibot  60 Aug  6 13:59 .
drwxr-x---. 4 eessibot eessibot 100 Aug  6 13:59 ..
-rw-r-----. 1 eessibot eessibot   0 Aug  6 13:59 foo

In other words: the issue here is that nobody nogroup is owning this. If I use mount=fuse, it is all owned by eessibot (and thus it can write).

Now, the question is: why does this work on some systems? It must have to do with how the user owning the CVMFS repo on the host is mapped into the container. I don't know enough about such mappings on how to resolve it. Can I map an individual UID into the container?

[casparvl]

Ok, on the host:

$ id
uid=72648(eessibot) gid=72109(eessibot) groups=72109(eessibot) ...
id cvmfs
uid=900(cvmfs) gid=900(cvmfs) groups=900(cvmfs)

My id:
uid=72648(eessibot) gid=72109(eessibot) groups=72109(eessibot) ...
Id of cvmfs user
uid=102(cvmfs) gid=102(cvmfs) groups=102(cvmfs)

I think this may be the cause of the issue. I might try to map the cvmfs user into the container and see if that works

[casparvl]

Doesn't seem to be possible. The biggest hammer I could find is --fakeroot, but even that doesn't help:

Apptainer> su
root@tcnXXX:~# touch /cvmfs/software.eessi.io/versions/foo
touch: cannot touch '/cvmfs/software.eessi.io/versions/foo': Permission denied

I'm going to leave it at this. It doesn't seem easily possible, my AI friend tells me that doing an overlay on top of a bind-mount can lead to unintended and strange interactions, and that is indeed what I'm observing. It's not fully clear why this works on some systems and fails on others. It probably has to do with the many different configurations you can set for singularity, or even the file systems, but... who knows which one.