raspberrypi-kernel-headers (1.20190925-2): Wrong exec format in scripts in buster #3279

livinghere · 2019-10-10T21:02:18Z

Is this the right place for my bug report?
yes

Describe the bug
/lib/modules/4.19.75-v7l+/build/scripts/basic/fixdep: Binary has wrong exec format

To reproduce
Fresh install of raspberrypi-kernel-headers (1.20190925-2) on a RPi4 with raspian buster. Executing the various tools for building kernel modules (i.e. wireguard) all complain about wrong exec format.

Expected behaviour
This did not happen on kernel version 4.19.66-v7l, so the bug is in the package of 4.19.75-v7l. Rebuild the packages with the correct target architecture.

Actual behaviour
see above

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
Pi4B
Which OS and version (cat /etc/rpi-issue)?
Raspberry Pi reference 2019-07-10
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 175dfb027ffabd4b8d5080097af0e51ed9a4a56c, stage2
Which firmware version (vcgencmd version)?
Sep 24 2019 17:34:59
Copyright (c) 2012 Broadcom
version cd3add54955f8fa065b414d8fc07c525e7ddffc8 (clean) (release) (start_cd)
Which kernel version (uname -a)?
Linux xxxxxx 4.19.75-v7l+ solved issue of mirroring screen after rotation. #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux
Logs

Additional context

The text was updated successfully, but these errors were encountered:

skyhisi · 2019-10-11T08:40:18Z

I've also seen this problem on all my Raspberry Pi's running buster when trying to build wireguard.

It can be worked around by applying the patches described here:
https://forum.loverpi.com/discussion/555/how-to-fix-dkms-error-bin-sh-1-scripts-basic-fixdep-exec-format-error

Based on that article it may be a cross compilation error.

HinTak · 2019-10-11T10:12:48Z

Yes, the files are all aarch64:

$ find /usr/src/ -type f -name 'fixdep' -exec file {} \;
/usr/src/linux-headers-4.19.75+/scripts/basic/fixdep: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=a1692535ab3b192a09bac27032606f64ec1815cb, not stripped
/usr/src/linux-headers-4.19.75-v7l+/scripts/basic/fixdep: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=a1692535ab3b192a09bac27032606f64ec1815cb, not stripped
/usr/src/linux-headers-4.19.75-v7+/scripts/basic/fixdep: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=a1692535ab3b192a09bac27032606f64ec1815cb, not stripped
/usr/src/linux-headers-4.19.75-v8+/scripts/basic/fixdep: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=a1692535ab3b192a09bac27032606f64ec1815cb, not stripped

XECDesign · 2019-10-11T10:49:45Z

Looking into it. Not sure what happened between raspberrypi/firmware#1155 was fixed and now.

I'll also add a check to make sure it doesn't happen again.

HinTak · 2019-10-11T10:55:11Z

The instruction in
https://forum.loverpi.com/discussion/555/how-to-fix-dkms-error-bin-sh-1-scripts-basic-fixdep-exec-format-error works for 4.19.75+ , 4.19.75-v7+ , 4.19.75-v7l+ . However, for 4.19.75-v8+, sudo make scripts start prompting for answers to scripts/kconfig/conf --syncconfig Kconfig.

scripts/kconfig/conf  --syncconfig Kconfig
*
* Restart config...
*
*
* General setup
*
...

For my needs I only need the other three, so this is just FYI.

XECDesign · 2019-10-11T10:56:48Z

Looks like the problem is that fixdep is built with HOSTCC. To add aarch64 kernel headers, I moved the build across to an aarch64 system - raspbian doesn't have an aarch64 cross-compiler.

I think the solution for now is to get rid of the headers matching kernel8.img. Since the headers package is for Raspbian and you won't be able to build out of tree modules for kernel8 anyway, that's probably not an issue.

Looking into whether there's a way to tell the build system to cross_compile fixdep as well first.

satmandu · 2019-10-11T14:05:50Z

For what it is worth I have had some luck with dealing with fixdep being built for AMD64 in ubuntu mainline kernel packages previously by installing qemu-user-binfmt , qemu-user, and libc6-amd64-cross like this:

sudo apt install libc6-amd64-cross qemu-user-binfmt qemu-user -y
sudo ln -rsf /usr/x86_64-linux-gnu/lib64 /lib64
sudo ln -rsf /usr/x86_64-linux-gnu/lib /lib/x86_64-linux-gnu

(Of course the headers in mainline ubuntu arm64 kernels are borked at the moment anyways... sigh.)

So something like this might work to let arm64 fixdep in headers work on a non arm64 system.

sudo apt install libc6-arm64-cross qemu-user-binfmt qemu-user -y
sudo ln -rsf /usr/aarch64-linux-gnu/lib /lib/aarch64-linux-gnu

Similarly, for an arm64 system (say booted with kernel8.img) with armhf fixdep in headers this might work:

sudo apt install libc6-armhf-cross qemu-user-binfmt qemu-user -y
sudo ln -rsf /usr/arm-linux-gnueabihf/lib /lib/arm-linux-gnueabihf

Frankly, it's annoying how much hackery is involved in cross-compiling the headers properly on a non-native system. Though I've managed to make it work using this spaghetti on ubuntu amd64 to get arm64 kernel packages built.:

#!/bin/bash
# Assumes you have standard ubuntu cross-compilers installed.
# For instance:
# sudo apt install  gcc-9-aarch64-linux-gnu \
#                cpp-9-aarch64-linux-gnu \
#                g++-9-aarch64-linux-gnu \
#                gcc-9-aarch64-linux-gnu-base \
#                libgcc-9-dev-arm64-cross \
#                libstdc++-9-dev-arm64-cross
mv_arch () {
        BUILDHOST_ARCH=$(uname -m)
        echo Replacing "${1}" with "${1}":"${2}"-cross.
        local dest_arch=${2}
        local dest_arch_prefix="${dest_arch}-linux-gnu-"
        local host_arch_prefix="${BUILDHOST_ARCH}-linux-gnu-"
        local file_out=$(file /usr/bin/"${1}")
        # Exit if dest arch file isn't available.
        [[ ! -f /usr/bin/${dest_arch_prefix}${1} ]] && PrintLog "Missing dest arch ${dest_arch_prefix}${1}" /tmp/compiler_setup.install.log && ragequit
        # If host arch backup file isn't available make backup.
        # This doesn't dereference symlinks!
        [[ ! -f /usr/bin/${host_arch_prefix}${1} && $(echo "${file_out}" | grep -m1 "${BUILDHOST_ARCH}") ]] && (
         cp /usr/bin/"${1}" /usr/bin/"${host_arch_prefix}""${1}"
         )
        if [[ $(echo "${file_out}" | grep -m1 "symbolic") ]]
            then
            rm /usr/bin/"${1}" && ln -s /usr/bin/"${dest_arch_prefix}""${1}" /usr/bin/"${1}"
        elif [[ -f /usr/bin/${dest_arch_prefix}${1} ]]
            then
            #cp ${dest_arch_prefix}${1} ${1}
            update-alternatives --install /usr/bin/"${1}" "${1}" /usr/bin/"${dest_arch_prefix}""${1}" 10
        fi
}
compiler_setup () {
    mv_arch ar aarch64
    mv_arch ld.bfd aarch64
    mv_arch ld aarch64
    mv_arch gcc-9 aarch64
    mv_arch g++-9 aarch64
    mv_arch cpp-9 aarch64
}
compiler_setup
# I now do the kernel compile thus:
#  cd ${workdir}/rpi-linux || exit 1
#  make -j${nprocs} CFLAGS=${CFLAGS} CCPREFIX=aarch64-linux-gnu- ARCH=arm64 \ 
#  CROSS_COMPILE=aarch64-linux-gnu- CROSS_COMPILE=aarch64-linux-gnu- \
#  LOCALVERSION=${LOCALVERSION} O=${workdir}/kernel-build/ bindeb-pkg

XECDesign · 2019-10-11T16:35:18Z

Should be fixed in 1.20190925+1-1

livinghere · 2019-10-11T19:03:46Z

Fixed.
Just tested with installing wireguard-0.0.20190913, DKMS building worked without errors.

Thank you all for the extremly quick response and successful fix!!

HinTak · 2019-10-14T13:19:28Z

The respeaker out-of-tree modules also work fine now. respeaker/seeed-voicecard#190

HinTak · 2019-10-15T15:42:28Z

https://forum.loverpi.com/discussion/555/how-to-fix-dkms-error-bin-sh-1-scripts-basic-fixdep-exec-format-error

FWIW, and for the record, I had another thought on that strange advice, and I think the patch itself is bogus. What actually mattered is the 2nd step - sudo make scripts - it forces native re-compilation of those bits.

Anyway, the new header package works. I just thought it is important to clarify what "voodoo" advices, like those, actually do...

XECDesign · 2019-10-15T15:49:19Z

I skimmed through it when you originally posted it, but couldn't make sense of it. If all it's doing is rebuilding 'scripts' (which fixdep is a part of), then it makes sense, but doesn't work around the cross-compiling issue.

A proper fix would look a bit more like this: https://bugs.launchpad.net/ubuntu/+source/linux-ti-omap4/+bug/666267/comments/16 or would involve patching the build system itself.

HinTak · 2019-10-15T16:39:56Z

You are not the only one puzzled by that strange piece of advice :-). I think that patch itself is bogus - especially given that all it does, is just adding two header files. Those I would assume, not used, since nothing else refer to them. What really happens is that the kernel's make system detects two new header files newer than "fixdep", and this causes "make scripts" to re-compile them on a native system. I agree with you that this does not fix cross-compiling. Playing with the timestamp like so: sudo touch tools/include/tools/*.h sudo make scripts instead of applying a bogus patch, would have done the job too, I think.

HinTak · 2021-02-04T00:41:56Z

@XECDesign hi, just curious how this was fixed in the end? I see the source package seems to be just repackaged binary tar balls. Am trying to crossbuild ubuntu kernel debs from x86_64 - it seems to work , except the same issue is cropping up (where fixdep targets x86_64...)

XECDesign · 2021-02-04T11:08:33Z

I build the full package on armhf and then reprepro identifies which dsc files need a rebuild, those files get pushed to appropriate build servers and the output it added back to reprepro. So they all end up being native builds.

HinTak · 2021-02-04T15:34:31Z

@XECDesign thanks. I guess touch ...; make scripts as a post install step (and appropriate deletes at preuninstall) is perhaps not bad. Fwiw, I tried https://bugs.launchpad.net/ubuntu/+source/linux-ti-omap4/+bug/666267/comments/16 and it does not work (can't find libc headers) - it seems to be part of a much larger patch set which got merged then reverted.

XECDesign · 2021-02-04T16:12:30Z

Yeah, I gave up on trying to get it to cross-compile the right 'host' binaries.

HinTak · 2021-02-04T16:29:09Z

I found the two linaro patches and discussions about them - they are quite large (touches about 4-5 files in non-trivial ways) and I kind-of agree they should not be merged. I wonder if it is possible to do post-install compile / pre-uninstall delete: this means the header package needs to depend on the compiler, but that's not bad since people installing the header likely wants to compile (out-of-tree drivers etc).

XECDesign · 2021-02-04T16:57:36Z

I'm not sure if there are additional files which would need to be added to the headers package for that to work. I'm fairly content with it as is for now.

HinTak · 2021-02-04T17:22:44Z

I think post-install compile + pre-install delete can probably work somewhat neatly (with the undesirable complication of the header package depending on build-essential or something), though I totally understand the desire not to spend too much time on fiddling with cross-compilation. I only realize quite recently that I have not done a full-kernel compilation for almost a decade! (ever since secure-boot became the norm, I think). Out-of-tree optional/additional module build is mostly what I have been doing, if I touch anything inside the kernel at all.

Btw, is the raspbian kernel package build log /procedure available on-line anywhere? I do mean the "deb package" build log, rather than the kernel ("make bcm2711_defconfig"). The purpose being to build packages which are drop-in compatible except for small and necessary tweaks/workarounds (like #4099 ).

XECDesign · 2021-02-04T17:52:05Z

It kind of built up from a much smaller package that was never intended to be used the way it currently is, so it's a bit of a hideous mess that nobody should look at.

I run ./debian/update.sh here https://github.com/RPi-Distro/firmware

You'll need to make sure submodules are updated and 'upstream' remote needs to point to https://github.com/raspberrypi/firmware.git.

HinTak · 2021-02-04T19:17:02Z

Thanks - I found https://github.com/RPi-Distro/firmware (from apt-get source ... I think) but wouldn't have noticed debian/update.sh. Btw, ubuntu has a slightly neater way of doing it, though it is somewhat constrained by its trying to have "one" tree across archs. By "one", I mean they have some horrid "if build-for-raspberrypi do-different-stuff" in their source tree too...

XECDesign · 2021-02-04T21:25:45Z

Yeah, reworking the whole thing and basing it on either the Debian or Ubuntu packages is something I've kept telling myself I'll do any day now for a while.

HinTak · 2021-02-04T22:08:11Z

Yeah, I think many people have a lot of misgivings about repository of binaries in general. I do that myself - for rarely changed/updated components - c# - which are neither available prepackaged nor too system-dependent - but has been told off by "purists" often.

So yeah, it would be nice to migrate to something closer to "mainstream".

The ubuntu raspberrypi pi kernel package is not difficult to cross-built (caveat the same fixdep issue here) - I even managed to do it on Fedora (a non-deb based system), without building any tools myself. It looks like I cannot do the same for raspbian, as git-buildpackage (which you use in your script) is not available on Fedora...

XECDesign · 2021-02-05T00:24:03Z

If you cross-build their package how do they handle the foreign binaries in headers issue?

HinTak · 2021-02-05T04:54:46Z

They don't. That goes back to my initial query here: I managed to finish ubuntu deb cross-build but ended up with the same problems - fixdep and a few things under scripts required for dkms ending up being the arch of the build host. Hence my question...

I am just going to put pushd header_dir touch ...; make scripts ; popd in the dkms set up for the time being. I think putting it in postinstall is neater.

HinTak · 2021-02-05T05:00:40Z

The two patches mentioned was still carried by linaro-ubuntu-packaging, but had not been upstreamed further:

http://git.linaro.org/ubuntu/linux-linaro-raring.git/commit/?h=linaro-ubuntu-packaging-3.7-aarch64&id=1688776d0e03cb3f41e488440e9ee217c02aa399

http://git.linaro.org/ubuntu/linux-linaro-raring.git/commit/?h=linaro-ubuntu-packaging-3.7-aarch64&id=86efc7ec96c2a4330102a33f2cc40ae318909c8b

HinTak · 2021-02-06T06:14:51Z

Fwiw, for 5.4 and/or ubuntu, neither of the approaches ("touch ... ; make scripts") or patching with KBUID_SCRIPTROOT patches works. The former ends with it prompting for config changes, the latter seems that the 3 binaries built for linaro-ubuntu then (fixdep, conf, mk_elfconfig) are not those needed for dkms (fixdep, recordmcount, modpost).

So I made a mess on my ubuntu pi install. But they do one thing better than raspbian: they allow multiple kernels installed, and symlink / update-initramfs between them; so it is possible to just copy the wrongly-built binaries from another header tree after experimenting.

The kbuid_scriptroot patches seems the proper way of doing it, except it needs a hell lot of update after almost a decade of bitrot...

satmandu · 2021-02-06T15:20:46Z

FYI, if you want a current headers package for the rpi-update 64-bit kernels, I'm using this:

https://gist.github.com/satmandu/a507c59d84737f6d29ff353395819d51

(I use it to install current 2.0.x zfs-dkms packages.)

HinTak · 2021-02-07T12:47:18Z

@satmandu Thanks for the URL, but I don't think the script explicitly caters for cross-compilation (which is what caused the issue here)?

HinTak mentioned this issue Oct 11, 2019

dkms build error against 4.19.75-* header, latest Buster respeaker/seeed-voicecard#190

Closed

holta mentioned this issue Oct 11, 2019

Segmentation fault with Gitea on a Raspi 4 #3271

Closed

This was referenced Oct 11, 2019

Gitea fails on Raspberry Pi iiab/iiab#1999

Closed

Kiwix install fails as part of install of current master [segmentation fault, due to Raspbian or Ubuntu 19.10 kernel on RPi] iiab/iiab#1993

Closed

twojciac mentioned this issue Oct 11, 2019

RPi 3B, Buster (4.19.75-v7+): Exec format error when Make gnab/rtl8812au#173

Closed

livinghere closed this as completed Oct 11, 2019

jancelin mentioned this issue Oct 24, 2019

Builder fails with "E: Version '1.20190718-1' for 'raspberrypi-kernel' was not found" hypriot/image-builder-rpi#313

Closed

XECDesign mentioned this issue Nov 25, 2019

Request for enhancement: 64-bit kernel headers RPi-Distro/repo#157

Closed

ZerBea mentioned this issue Dec 8, 2019

Error installing dkms in Raspberry Pi 4 aircrack-ng/rtl8812au#506

Open

HinTak mentioned this issue Mar 19, 2020

Remove Kernel Downgrade respeaker/seeed-voicecard#209

Closed

3 tasks

raspberrypi-kernel-headers (1.20190925-2): Wrong exec format in scripts in buster #3279

raspberrypi-kernel-headers (1.20190925-2): Wrong exec format in scripts in buster #3279

Comments

livinghere commented Oct 10, 2019

skyhisi commented Oct 11, 2019

Uh oh!

HinTak commented Oct 11, 2019

Uh oh!

XECDesign commented Oct 11, 2019

Uh oh!

HinTak commented Oct 11, 2019

Uh oh!

XECDesign commented Oct 11, 2019

Uh oh!

satmandu commented Oct 11, 2019

Uh oh!

XECDesign commented Oct 11, 2019

Uh oh!

livinghere commented Oct 11, 2019

Uh oh!

HinTak commented Oct 14, 2019

Uh oh!

HinTak commented Oct 15, 2019

Uh oh!

XECDesign commented Oct 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HinTak commented Oct 15, 2019 via email

Uh oh!

HinTak commented Feb 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XECDesign commented Feb 4, 2021

Uh oh!

HinTak commented Feb 4, 2021

Uh oh!

XECDesign commented Feb 4, 2021

Uh oh!

HinTak commented Feb 4, 2021

Uh oh!

XECDesign commented Feb 4, 2021

Uh oh!

HinTak commented Feb 4, 2021

Uh oh!

XECDesign commented Feb 4, 2021

Uh oh!

HinTak commented Feb 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XECDesign commented Feb 4, 2021

Uh oh!

HinTak commented Feb 4, 2021

Uh oh!

XECDesign commented Feb 5, 2021

Uh oh!

HinTak commented Feb 5, 2021

Uh oh!

HinTak commented Feb 5, 2021

Uh oh!

HinTak commented Feb 6, 2021

Uh oh!

satmandu commented Feb 6, 2021

Uh oh!

HinTak commented Feb 7, 2021

Uh oh!

XECDesign commented Oct 15, 2019 •

edited

Loading

HinTak commented Feb 4, 2021 •

edited

Loading

HinTak commented Feb 4, 2021 •

edited

Loading