npm hangs on linux/s390x containers

[hardillb]

Environment

• Platform: linux/s390x
• Docker Version: 24.0.6, build ed223bc
• Node.js Version: 18
• Image Tag:18-alpine

Expected Behavior

npm install runs and packages are installed.

Current Behavior

Trying to build a container on the linux/s309x platform hangs running npm install with npm consuming 100% CPU.

Previous builds complete in less than 5mins, current build has been running for over an hour

We are building the https://github.com/node-red/node-red-docker container with

docker buildx build --platform linux/s390x --file .docker/Dockerfile.alpine --build-arg NODE_VERSION=18 .

Possible Solution

Steps to Reproduce

• Check out https://github.com/node-red/node-red-docker
• cd node-red-docker
• run docker buildx build --platform linux/s390x --file .docker/Dockerfile.alpine --build-arg NODE_VERSION=18 .

Additional Information

Same thing is happening with 14-alpine and 16-alpine tags

I'm hitting this both locally and in a GH Action, both of which use Qemu to support building for alternate architectures.

[tyranron]

I have similar issue (see Dockerfile).

I wonder whether the problem of #1798 and #1829 finally snuck into 18 and earlier images.

[sxa]

Interesting. I've just fired up the docker image (node:16-alpine and node:18-alpine) on a real s390x system and npm seems to install without any problems. Which would lead us to perhaps something specific to qemu or the docker version in use (Mine is Docker version 24.0.5, build 24.0.5-0ubuntu1~22.04.1)

[sxa]

Just tried with your dockerfile - went through without problems:
build18.log.gz
Command: docker build --file .docker/Dockerfile.alpine --build-arg NODE_VERSION=18 . 2>&1 | tee build18.log

[hardillb]

Which does appear to point to this possibly being a qemu based problem. I know my laptop got a recent set of qemu packages, but not sure what would be needed to debug this. Any pointers would be helpful

[tyranron]

@hardillb setup-qemu-action uses onistiigi/binfmt Docker image for installing QEMU binaries. I think other versions like 6.1.0 or master could be tried to "resolve" this at least on GitHub Actions.

[hardillb]

master doesn't appear to fix it for me, testing 6.1.0

[hardillb]

no joy with qemu-v6.1.0 either so this may be a NodeJS + Qemu issue

[hardillb]

OK, while this appears to be limited to when running builds using qemu, this is going to be the default way 99% of CI builds run that target s390x, so I think we still need to track this down, even if it's just to raise a sensible upstream issue against qemu.

• What debug options can I enable to try and get some useful debug information here?
• Would trying to connect GDB to the spinning process help?

[tyranron]

@hardillb seems like after moby/buildkit#1516 we may omit using setup-qemu-action, because BuildKit supports QEMU emulation out-of-the-box. Even more, judging by onistiigi/binfmt Docker image tags, newer version of QEMU are released for buildkit- images only. The last one is 7.1.0.

However, for my repository the result is still the same, no matter which version is used: 6.0.0, 6.1.0, 6.2.0, 7.0.0, 7.1.0 or master.

[tyranron]

@hardillb in my case, the problem seems to be related to Linux only, somehow. I was able to resolve the issue just by switching to macos-latest runner for archs where the build stucks.

I will try this workaround for #1798 too, and will report the results.

[hardillb]

@tyranron did you get any joy using the docker.io/ prefix on the base containers?

If it is the qemu but https://gitlab.com/qemu-project/qemu/-/issues/1729 then hopefully it gets fixed soon.

[tyranron]

@hardillb

did you get any joy using the docker.io/ prefix on the base containers?

These are the same images, no?

I will try this workaround for #1798 too, and will report the results.

Building under macos-latest runner didn't work out for Node.js 20, but for 18 it fixed my problem.

[hardillb]

This may not be the same as the other qemu bug as it's not calling mremap.

I ran the following command:

docker run --platform linux/s390x -it --cap-add=SYS_PTRACE -e QEMU_STRACE=true -e QEMU_LOG_FILENAME=qemu.log -v ./qemu.log:/qemu.log --rm node:18-alpine npm install node-red:3.1.0

and got the following strace:

1 clock_gettime(CLOCK_MONOTONIC,0x0000004007b62cd8) = 0 ({tv_sec = 2223689,tv_nsec = 708242416})
1 clock_gettime(CLOCK_MONOTONIC,0x0000004007b62db8) = 0 ({tv_sec = 2223689,tv_nsec = 708269945})
1 clock_gettime(CLOCK_MONOTONIC,0x0000004007b62db8) = 0 ({tv_sec = 2223689,tv_nsec = 708530498})
1 clock_gettime(CLOCK_MONOTONIC,0x0000004007b62eb0) = 0 ({tv_sec = 2223689,tv_nsec = 708556031})
1 clock_gettime(CLOCK_MONOTONIC,0x0000004007b62eb0) = 0 ({tv_sec = 2223689,tv_nsec = 708593060})
1 munmap(0x00000040101ef000,57344) = 0
1 clock_gettime(CLOCK_MONOTONIC,0x0000004007b63028) = 0 ({tv_sec = 2223689,tv_nsec = 708653152})
1 socket(PF_NETLINK,SOCK_RAW|SOCK_CLOEXEC,NETLINK_ROUTE) = 24
1 sendto(24,275007277688,20,0,0,0) = 20
1 recvfrom(24,275007277688,8192,64,0,0) = 2880

qemu.log

[hardillb]

This looks to be spinning trying to receive data from the network. How do we move this forward?