Problems with building a 4.8/4.9 kernel in 64bit mode, bad file format for "Image"

[DieterReuter]

I try to build a 64bit kernel for RPi3. Building is just easy and straight forward, but the resulting Image kernel image cannot be booted at all. It seems the generated file format is just wrong, but I don't have any glue how to resolve it.

This kernel is my build result which won't boot at all:

$ ls -al Image
-rwxr-xr-x  1 dieter  staff  9525760 Jan 10 23:40 Image

$ file Image
Image: MS-DOS executable, MZ for MS-DOS

$ xxd Image | head -10
00000000: 4d5a 0091 ff3f 1e14 0000 0800 0000 0000  MZ...?..........
00000010: 00c0 9d00 0000 0000 0a00 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 4152 4d64 4000 0000  ........ARMd@...
00000040: 5045 0000 64aa 0200 0000 0000 0000 0000  PE..d...........
00000050: 0100 0000 a000 0602 0b02 0214 00b0 9d00  ................
00000060: 0000 0000 0000 0000 204f 7900 0010 0000  ........ Oy.....
00000070: 0000 0000 0000 0000 0010 0000 0002 0000  ................
00000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000090: 00c0 9d00 0010 0000 0000 0000 0a00 0000  ................

If I compare it with a running kernel from this tarball http://www.tom-yam.or.jp/rpi3/rpi3-arm64-debian-20160414.img.xz from this forum thread, I get the following file format:

$ ls -al Image
-rwxr-xr-x  1 dieter  staff  9513920 Jan 10 17:55 Image

$ file Image
Image: data

$ xxd Image | head -10
00000000: 1000 0014 0000 0000 0000 0800 0000 0000  ................
00000010: 0040 9500 0000 0000 0200 0000 0000 0000  .@..............
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 4152 4d64 0000 0000  ........ARMd....
00000040: 0900 0094 e703 0094 18fc ff90 2104 0094  ............!...
00000050: 0d00 0094 7b0b 0058 be2e 0090 de03 0091  ....{..X........
00000060: 7853 0014 f503 00aa 2045 00d0 0000 0291  xS...... E......
00000070: 1504 00a9 020c 01a9 bf3f 03d5 0180 0091  .........?......
00000080: 604e 0014 794a 00f0 9a4a 00d0 fb03 1eaa  `N..yJ...J......
00000090: e003 19aa 410b 4091 5a4e 0094 e003 19aa  ....A.@.ZN......

Both kernels have rougly the same size, but different file format.

Any help or pointer greatly appreciated.

And here are my building steps, just to make sure there's no wrong configuration:

# Ubuntu 16.10
apt-get update
apt-get install -y bc build-essential gcc-aarch64-linux-gnu git

# Clone kernel source
git clone --depth 1 https://www.github.com/raspberrypi/linux -b rpi-4.8.y
cd ./linux

# Compile kernel
export CROSS_COMPILE=aarch64-linux-gnu-
export ARCH=arm64
make bcmrpi3_defconfig
make -j 8

[DieterReuter]

Maybe I'm totally wrong and my kernel image is 100% ok, and the problem is just the way I'm booting the kernel through U-Boot.

Just installed and tested openSUSE from this SD image openSUSE-Leap42.2-ARM-JeOS-raspberrypi3.aarch64-2017.01.07-Build1.1.raw.xz. Their kernel image gives me:

$ file Image-4.4.36-8-default
Image-4.4.36-8-default: MS-DOS executable, MZ for MS-DOS

$ xxd Image-4.4.36-8-default | head -10
0000000: 4d5a 0091 ff03 0014 0000 0800 0000 0000  MZ..............
0000010: 00c0 ec00 0000 0000 0200 0000 0000 0000  ................
0000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000030: 0000 0000 0000 0000 4152 4d64 4000 0000  ........ARMd@...
0000040: 5045 0000 64aa 0200 0000 0000 0000 0000  PE..d...........
0000050: 0100 0000 a000 0602 0b02 0214 00b0 ec00  ................
0000060: 0000 0000 0000 0000 80aa bd00 0010 0000  ................
0000070: 0000 0000 0000 0000 0010 0000 0002 0000  ................
0000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000090: 00c0 ec00 0010 0000 0000 0000 0a00 0000  ................

Does anybody have some hints for me how to use boot the 4.8 kernel with U-Boot? Any documentation where I can easily follow all the required steps? Thanks!

[Electron752]

I haven't tried u-boot in awhile, but I think the trick is to rebase the kernel image to use a different load address then the default. The tool you want to use is mkimage and it's included in the "u-boot-tools" package in debian sid on my PC.

Essentially, you rename the u-boot loader to be kernel8.img and then have u-boot load the image you create with mkimage. You need to create a u-boot script file to manually load the correct device tree as part of the boot process.

BTW, I understand that a version of u-boot is available that has UEFI support. I've heard of people using u-boot to load grub2 which then loads the Linux kernel. The point is to get the fancy menus of grub to work.

[DieterReuter]

@Electron752 Thanks for the infos. I just found out the kernel will be created by default with UEFI support, that's the reason for the different file format. As soon as create the kernel w/o EFI then the file format looks correct. But it doesn't boot.

I'm already using the latest mainline U-Boot v2017.01-rc3 and did verified my settings with successfully booting the 4.5.0 kernel I mentioned above.

[Electron752]

Yes, I think the image format and boot process have changed alot since last april(2016). Back then, you had to cat a special asm stub to the start of the kernel binary. That isn't necessary anymore and I don't think it's supported anymore directly by the RPI firmware/bootloader.

Did you check out the mkimage command from u-boot tools? Last time I had u-boot working, it was necessary to run this step to generate a rebased kernel in the special u-boot format. I think by default the load address of the u-boot loader conflicts with the default load address of the kernel image. That's why it's necessary to rebase the kernel image. You also need to modify your u-boot loader script to load the 64 bit version of the device tree. Since you are going through u-boot, it isn't loaded by default. If you try to use the 32 bit version of the device tree, you will not get very far into the boot process. Probably just a black screen.

I can probably do a complete writeup on how to use u-boot, but I'm a bit busy ATM and the process of using u-boot is a bit complex. But I think the info I've given should point in the correct direction.

I am curious why you want to use u-boot in the first place. For simple booting, it doesn't add much value to the default loader that ships with the RPI. What it does give is more advanced boot options such as network booting and UEFI support.

[DieterReuter]

Honestly my goal is just to get a 64bit kernel running on the RPi, w/ or w/o U-Boot doesn't really matter. But with U-Boot I do have some more control in the dev process and later I can use PXE boot to speed up the kernel configuration and testing process.

BTW, the U-Boot booti command takes care about kernel relocation, so mkimage is not necessary.

Here is my process so far to use U-Boot:

1. U-Boot boot script:

$ cat boot.txt
fatload mmc 0:1 ${fdt_addr_r} bcm2837-rpi-3-b.dtb
fatload mmc 0:1 ${kernel_addr_r} Image
setenv bootargs root=/dev/mmcblk0p2 rootfstype=ext4 \
  cgroup_enable=cpuset cgroup_enable=memory swapaccount=1 elevator=deadline \
  fsck.repair=yes rootwait rw console=ttyS0,115200 kgdboc=ttyS0,115200
booti ${kernel_addr_r} - ${fdt_addr_r}

compile boot script, U-Boot for RPi3 is looking for default script "boot.scr.uimg"

$ mkimage -A arm64 -O linux -T script -C none -d boot.txt $SDCARD/boot.scr.uimg

2. Copy U-boot

$ cp u-boot.bin $SDCARD/kernel8.img

3. Enable UART to get boot log messages and to use the U-Boot shell

$ cat config.txt
enable_uart=1
$ cp config.txt $SDCARD/

4. Content of the SD card FAT partition

$ ls -al $SDCARD
-rwxr-xr-x   1 dieter  staff  9535496 Jan 12 13:19 Image
-rw-r--r--   1 dieter  staff     7500 Jan 12 13:19 bcm2837-rpi-3-b.dtb
-rw-r--r--   1 dieter  staff      452 Jan 12 13:22 boot.scr.uimg
-rw-r--r--   1 dieter  staff    50832 Jan 11 21:43 bootcode.bin
-rw-r--r--   1 dieter  staff       14 Jan 12 13:22 config.txt
-rw-r--r--   1 dieter  staff     6630 Jan 11 21:43 fixup.dat
-rwxr-xr-x   1 dieter  staff   368848 Jan 12 13:22 kernel8.img
-rw-r--r--   1 dieter  staff  2828260 Jan 11 21:43 start.elf

5. Final result

U-Boot 2017.01-rc3-bee42 (Jan 12 2017 - 10:42:45 +0000)

DRAM:  944 MiB
RPI 3 Model B (0xa02082)
MMC:   bcm2835_sdhci: 0
reading uboot.env

** Unable to read "uboot.env" from mmc0:1 **
Using default environment

In:    serial
Out:   lcd
Err:   lcd
Net:   Net Initialization Skipped
No ethernet found.
starting USB...
USB0:   Core Release: 2.80a
scanning bus 0 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
       scanning usb for ethernet devices... 1 Ethernet Device(s) found
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr.uimg
reading /boot.scr.uimg
443 bytes read in 10 ms (43 KiB/s)
## Executing script at 02000000
reading bcm2837-rpi-3-b.dtb
7500 bytes read in 18 ms (406.3 KiB/s)
reading Image
9535496 bytes read in 615 ms (14.8 MiB/s)
## Flattened Device Tree blob at 00000100
   Booting using the fdt blob at 0x000100
   Loading Device Tree to 000000003ab28000, end 000000003ab2cd4b ... OK

Starting kernel ...

Here it stops forever with my own kernels, I've tested 4.8 and 4.9.

[Electron752]

I'll see what I can do it get u-boot working either later today or possibly the weekend.

In the meantime, I have a version of 4.9 that works now. I've created pull request #1790 to update the downstream tree.

If you want it sooner, you can get it off my github page at http://www.github.com/Electron752/linux branch rpi-4.9.y+rpi364.

Booting without u-boot is very easy. Just build like you have done and copy:
arch/arm64/boot/Image -> /boot/kernel8.img
arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb -> /boot/bcm2710-rpi-3-b.dtb
arch/arm64/boot/dts/overlays/* -> /boot/overlays/

This works for me without changing anything in config.txt. But if it doesn't work for you, you may want to try adding:
(config.txt)
device_tree=bcm2710-rpi-3-b.dtb

[DieterReuter]

Thanks, will try your 4.9 repo later!

[DieterReuter]

This kernel version works, thanks!!!
🎉

[pelwell]

The latest rpi-4.9.y contains #1790.

[Electron752]

@DieterReuter
Glad to hear that it works.

I might delete the temporary branch off my github page in a few days. I generally just use my page as a staging area.

[DieterReuter]

I close this issue, because it's resolved for me now.

[02350316]

hi DieterReuter:
i meet the same question,use u-boot boot the kernel,and only get this:starting kernel.......
may i ask : final you boot succses,do you use u-boot,or not?

thanks

[DieterReuter]

@02350316 I switched to kernel 4.9 repo and don't use u-boot now. So, you just need a kernel with serial output, then it should work. And there has to be the correct .dtb file in place.