how to build for powerpc openwrt musl libc 1.2.3 device featuring e500v1 core ?

[thomas725]

For the last few weeks with the help of @Emilgardis I've tried to get rust compiled for my powerpc openwrt device.

TP-Link TL-WDR4900 v1.x

See also: #1349

Currently we're testing those images created by @Emilgardis:

• http://ghcr.io/emilgardis/powerpc-unknown-linux-musl-cross:test
• http://ghcr.io/emilgardis/powerpc-unknown-linux-muslsf-cross:test

We managed to get the first one of those to produce debug build using:

RUSTFLAGS="-C target-feature=-crt-static" cross +nightly build --target powerpc-unknown-linux-musl

but since the resulting binary is 15mb big I'd need a release build to actually test if it can run on my device, which failed to build so far:

$ RUSTFLAGS="-C target-feature=-crt-static" cross -Zbuild-std=std,panic_abort +nightly build --target powerpc-unknown-linux-musl --release
[cross] warning: using newer rustc `1.75.0-nightly (2f1bd0729 2023-10-27)` for the target. Current active rustc on the host is `rustc 1.73.0 (cc66ad468 2023-10-03)`.
 > Update with `rustup update`
   Compiling compiler_builtins v0.1.101
   Compiling core v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core)
   Compiling libc v0.2.149
   Compiling cc v1.0.79
   Compiling memchr v2.5.0
   Compiling std v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std)
   Compiling unwind v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/unwind)
   Compiling rustc-std-workspace-core v1.99.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-core)
   Compiling alloc v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc)
   Compiling cfg-if v1.0.0
   Compiling adler v1.0.2
   Compiling rustc-demangle v0.1.23
   Compiling rustc-std-workspace-alloc v1.99.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_unwind)
   Compiling panic_abort v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_abort)
   Compiling gimli v0.28.0
   Compiling miniz_oxide v0.7.1
   Compiling std_detect v0.1.5 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/stdarch/crates/std_detect)
   Compiling object v0.32.0
   Compiling hashbrown v0.14.2
   Compiling addr2line v0.21.0
   Compiling proc_macro v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/proc_macro)
   Compiling my-project v0.1.0 (/home/user/Development/git/my-project)
error[E0463]: can't find crate for `panic_abort`

For more information about this error, try `rustc --explain E0463`.
error: could not compile `my-project` (bin "my-project") due to previous error
[cross] warning: rust-std is not available for powerpc-unknown-linux-musl
[cross] note: you may need to build components for the target via `-Z build-std=<components>` or in your cross configuration specify `target.powerpc-unknown-linux-musl.build-std`
              the available components are core, std, alloc, and proc_macro

[thomas725]

commenting out the panic= property in this Cargo.toml block

[profile.release]
opt-level = 'z'   # Optimize for size
lto = true        # Enable link-time optimization
codegen-units = 1 # Reduce number of codegen units to increase optimizations
#panic = 'abort'   # Abort on panic
strip = true # Strip symbols from binary*

makes this release build command work too:

CROSS_CONTAINER_ENGINE=podman RUSTFLAGS="-C target-feature=-crt-static" cross +nightly build --target powerpc-unknown-linux-musl --release

and results in a 321kb big binary:

ELF 32-bit MSB pie executable, PowerPC or cisco 4500, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-powerpc.so.1, stripped

Running that binary on my openwrt router sadly doesn't print "hello world" as expected, but instead:

-ash: /bin/my-project: not found

I guess that's because it's looking for the first while we only have the second on device:

root@openwrt:~# ls -alih /lib/ld-musl-powerpc.so.1
ls: /lib/ld-musl-powerpc.so.1: No such file or directory
root@openwrt:~# ls -alih /lib/ld-musl-powerpc-sf.so.1 
    305 lrwxrwxrwx    1 root     root           7 Oct 15  2022 /lib/ld-musl-powerpc-sf.so.1 -> libc.so

[thomas725]

ohhh, but it actually is compatible! running

ln -s /lib/libc.so /lib/ld-musl-powerpc.so.1

makes my binary print the expected output!

root@openwrt:~# /bin/my-project
hello world

[thomas725]

@Emilgardis Can you help me translate our success into something I can put into a .gitlab-ci.yml task?
I'm reading #273 but am a bit confused in how complex that task seems to be. Do I really need to run docker in docker, can't I just use the build image directly?

[thomas725]

also, will this solution blow up as soon as I do anything that uses float-point operations? since my musl-libc is actually soft-float but the compiler seems to think we have hardware-float?

Can you make the second = soft-float image public so I can use that one instead? Also can you merge what we managed to do into the regular cross-rs infrastructure?

[Emilgardis]

Awesome!

Cross works by running in/starting a container, so unfortunately container in container is needed. If you want you can skip cross and just emulate what cross does but without cross. Shouldn't be too hard, just copy the dockerfile I've made and supply all the needed files and use that as the base image for the runner, then also install rustup and you should be all good :)

If you do want to use cross for Gitlab Runners, recommended usage is via CROSS_REMOTE now. see https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci

[Emilgardis]

also, could you try adding a float operation to the binary, interested if the device actually can do hard floats.

something like

println!("{:?}", (1.0f32/10.0).sin());

[thomas725]

no, doesn't seem like it can:

hello world
Illegal instruction

running in dev environment this instead prints:

hello world
0.09983342

I will have a look at https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci tomorrow and see if I can make that work, thanks for your great support, you're amazing :) gotta get some sleep now.

[thomas725]

Okey, so I don't think what's described at https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci can work for me, since I'm not hosting a gitlab instance myself but just using the public gitlab.com instance which includes 400 free CI minutes myself - but I don't think (or know how) I can access a docker host from there like described at tcp://docker:2375/

I'm considering using github instead, but the section about github says it need's to be reworked because actions-rs is unmaintained.

[Emilgardis]

You should be able to use tcp://docker:2375/ just fine, doesnt matter if you host the instance yourself or not.

So, for only using softfloat, you can do RUSTFLAGS="-C target-feature=-crt-static,-hard-float" and also use the muslsf image instead.

Another thing you could do is use a custom target, this way you don't have to specify any rustflags via env.

#Cross.toml
[target."powerpc-unknown-linux-muslsf.json"]
image = "ghcr.io/emilgardis/powerpc-unknown-linux-muslsf-cross:test"
build-std = true

# powerpc-unknown-linux-muslsf.json
{
    "arch": "powerpc",
    "crt-objects-fallback": "musl",
    "crt-static-default": false,
    "crt-static-respected": true,
    "data-layout": "E-m:e-p:32:32-Fn32-i64:64-n32",
    "dynamic-linking": true,
    "env": "musl",
    "has-rpath": true,
    "has-thread-local": true,
    "linker-flavor": "gnu-cc",
    "llvm-target": "powerpc-unknown-linux-musl",
    "max-atomic-width": 32,
    "os": "linux",
    "position-independent-executables": true,
    "pre-link-args": {
        "gnu-cc": [
            "-m32"
        ],
        "gnu-lld-cc": [
            "-m32"
        ]
    },
    "relro-level": "full",
    "stack-probes": {
        "kind": "inline"
    },
    "supported-split-debuginfo": [
        "packed",
        "unpacked",
        "off"
    ],
    "target-endian": "big",
    "target-family": [
        "unix"
    ],
    "target-mcount": "_mcount",
    "target-pointer-width": "32",
    "features": "-hard-float"
}

cross +nightly build --release --target powerpc-unknown-linux-muslsf.json

[Emilgardis]

looking further into the MPC8540, you might be fine with SPE. this comment is a bit concerning though

The following CPUs are known to not working with this port:

MPC8540 (for this one, you can use Debian's regular powerpc port with activated FPU emulation in the kernel)

#Cross.toml
[target."powerpc-unknown-linux-muslspe.json"]
image = "ghcr.io/emilgardis/powerpc-unknown-linux-muslsf-cross:test"
build-std = true

# powerpc-unknown-linux-muslspe.json
{
    "arch": "powerpc",
    "crt-objects-fallback": "musl",
    "crt-static-default": false,
    "crt-static-respected": true,
    "data-layout": "E-m:e-p:32:32-i64:64-n32",
    "dynamic-linking": true,
    "env": "musl",
    "has-rpath": true,
    "has-thread-local": true,
    "linker-flavor": "gnu-cc",
    "llvm-target": "powerpc-unknown-linux-muslspe",
    "max-atomic-width": 32,
    "os": "linux",
    "position-independent-executables": true,
    "pre-link-args": {
        "gnu-cc": [
            "-m32 -mspe"
        ],
        "gnu-lld-cc": [
            "-m32 -mspe"
        ]
    },
    "relro-level": "full",
    "stack-probes": {
        "kind": "inline"
    },
    "supported-split-debuginfo": [
        "packed",
        "unpacked",
        "off"
    ],
    "target-endian": "big",
    "target-family": [
        "unix"
    ],
    "target-mcount": "_mcount",
    "target-pointer-width": "32",
    "features": "-fpu,+spe"
}

[thomas725]

hmm, something with that custom target json didn't work as hoped:

$ cross +nightly build --target powerpc-unknown-linux-muslsf.json --release
[cross] warning: using newer rustc `1.75.0-nightly (2f1bd0729 2023-10-27)` for the target. Current active rustc on the host is `rustc 1.73.0 (cc66ad468 2023-10-03)`.
 > Update with `rustup update`
   Compiling my-project v0.1.0 (/home/user/Development/git/my-project)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin" VSLANG="1033" "cc" "-m32" "/tmp/rustctgSOPN/symbols.o" "/target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3.my_project.2be8c3d152745613-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/target/powerpc-unknown-linux-muslsf/release/deps" "-L" "/target/release/deps" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslsf/lib" "-Wl,-Bstatic" "/target/powerpc-unknown-linux-muslsf/release/deps/libcompiler_builtins-ef09095d5d60da69.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslsf/lib" "-o" "/target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
  = note: /usr/bin/ld: cannot find Scrt1.o: No such file or directory
          /usr/bin/ld: cannot find crti.o: No such file or directory
          ...
          /usr/bin/ld: /target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3.my_project.2be8c3d152745613-cgu.0.rcgu.o: relocations in generic ELF (EM: 20)
          /usr/bin/ld: /target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3.my_project.2be8c3d152745613-cgu.0.rcgu.o: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status

[thomas725]

and the spe version fails like this:

error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin" VSLANG="1033" "cc" "-m32 -mspe" "/tmp/rustcGLJTbP/symbols.o" "/target/powerpc-unknown-linux-muslspe/release/deps/my_project-5fc72a3857ce6d3c.my_project.d68e5b2406453db7-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/target/powerpc-unknown-linux-muslspe/release/deps" "-L" "/target/release/deps" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslspe/lib" "-Wl,-Bstatic" "/target/powerpc-unknown-linux-muslspe/release/deps/libcompiler_builtins-5290246e7a0b36d3.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslspe/lib" "-o" "/target/powerpc-unknown-linux-muslspe/release/deps/my_project-5fc72a3857ce6d3c" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
  = note: cc: error: unrecognized command line option '-m32 -mspe'

[thomas725]

okey so the binary resulting from this command:

RUSTFLAGS="-C target-feature=-crt-static,-hard-float" CROSS_CONTAINER_ENGINE=podman cross +nightly build --target powerpc-unknown-linux-musl --release

gives good float output:

hello world
0.09983342

even though it does output this warning during building:

warning: unknown feature specified for `-Ctarget-feature`: `hard-float`
  |
  = note: it is still passed through to the codegen backend
  = help: consider filing a feature request

[Emilgardis]

that warning is what I'm discussing in rust-lang/rust#117347

for the powerpc-unknown-linux-muslsf.json and powerpc-unknown-linux-muslspe.json targets, forgot that we also need to specify the linker

i'm not certain what the envvar would be called but you can set the linker in .cargo/config.toml

should be powerpc-linux-muslsf-gcc