Use Case for Build Script Inversion: MicroZig

[ikskuh]

This is a response to a comment from #18778:

direct @import in build scripts

I looked into making this work for lazy dependencies. It's possible, but it has some downsides:

• it requires generating a zig source file per dependency instead of one dependencies.zig file for the whole tree. I don't know how much of a problem this is in reality, perhaps it's more distasteful in theory than in practice.
• It requires some clunky API in the build.zig script - an explicit check for the existence of a dependency followed by the @import, or something like that.

Maybe I could find a more satisfactory solution if I keep poking at it.

That said, this feature was always meant for actual build script dependencies, it was not intended for inversion of control and having the dependency take over the build script logic for the parent package. I think users should try to get away from doing that, and I'm happy to work together on coming up with a satisfactory API that is based on dependencies exposing things rather than doing build logic.

MicroZig currently heavily relies on this kind of build script inversion, as it tries to provide a similar experience to the user as a regular desktop build script does.

Tasks that need to be done for building a single executable for an embedded task usually require the following steps:

1. Generate/write fitting linker script that is basically the same for all MCUs except for a hand full of offsets and lengths.
2. (optionally) Compile a bootloader
3. Compile the main application. This depends on modules for:
   • Application (this is what the user wants to write)
   • (optional) Driver Packages (which the user wants to include)
   • Boot Logic (depending on the previously compiled bootloader, if any)
   • CPU support
   • MCU support
   • (optional) Board support
   • (optional) Hardware Abstraction Layer
4. (optional) Patch the resulting ELF file to insert checksums, boot marks, …
5. (optional) Convert the ELF file from step 4. into another format like UF2, Intel Hex, DFU, …
   • Depends on external tooling and libraries like ZIP, …

Usually the build script also has features to flash the target CPU, which also requires additional tooling and build steps.

The whole idea of MicroZig is to offload the boilerplate into the MicroZig package and let the user focus on what they care about:

Application code and optional driver/package dependencies

Everything else is basically 100% repetetive between projects and doesn't differ between basically all build processes.

That said, this feature was always meant for actual build script dependencies, it was not intended for inversion of control and having the dependency take over the build script logic for the parent package. I think users should try to get away from doing that, and I'm happy to work together on coming up with a satisfactory API that is based on dependencies exposing things rather than doing build logic.

Right now, MicroZig has over 2000 lines of code for build automation, which don't have much "logic" inside them glueing different steps and modules together to solve the tasks above.

I would rather not have to force every user to have something like 1500 LOC of build script in their embedded projects, because this makes them really hard to maintain, as copy-paste errors start to emerge over time.

@andrewrk, i'm happy to find a solution to make build scripts/automation simpler in the future that could work without having build script inversion, but i'm not sure how to do that.

My goal is that we can keep the build scripts as simple as they are right now:

const std = @import("std");
const MicroZig = @import("microzig-build");

pub fn build(b: *std.Build) void {
    const microzig = MicroZig.createBuildEnvironment(b, .{});

    const show_targets_step = b.step("show-targets", "Shows all available MicroZig targets");
    show_targets_step.dependOn(microzig.getShowTargetsStep());

    const optimize = b.standardOptimizeOption(.{});
    const target_name = b.option([]const u8, "target", "Select the target to build for.") orelse "board:mbed/lpc1768";

    const target = microzig.findTarget(target_name).?;

    const firmware = microzig.addFirmware(b, .{
        .name = "blinky",
        .target = target,
        .optimize = optimize,
        .source_file = .{ .path = "src/empty.zig" },
    });

    microzig.installFirmware(b, firmware, .{});
}

There are some questionable design decisions in there, but those could be changed to include more boilerplate, but would make some dependencies more explicit.

[castholm]

It would be useful to define exactly what "inversion of control" means in the context of build scripts, because I'm not sure I understand what it means here.

I completely get if something horrid like @import("lib").setUpABazillionBuildSteps(b, lib_dep, target, optimize, exe, so, main_mod, secondary_mod) that orchestrates more or less the entire parent package build step graph would be considered an anti-pattern, but I fail to see what the problem with something like const sprite_sheet: std.Build.LazyPath = @import("lib").createSpriteSheet(b, &sprite_sources, .{}) (which you might imagine compiles an artifact, runs it while passing a set of source files to it, and then return its generated output file) would be.

I'll chime in with a similar use case in my (WIP) bindings/wrapper library for the Playdate console, which also has some very specific constraints (embedded target, must use a specific linker script, must emit relocs, bundling binaries/assets using a proprietary SDK, debugging using a simulator, etc.) which require a lot of minute configuring and composing of build steps that would be very inconvenient to have to copy/paste into every consuming project.

Through the power of @importing build scripts I've been able to expose a build API that lets a user configure their project in a simple and familiar way that is very similar to native Zig build steps and which I feel does not at all conflict with any of Zig's guiding philosophies:

const std = @import("std");
const playdate = @import("playdate");

pub fn build(b: *std.Build) void {
    const pdex_targets = playdate.standardExecutableTargetOptions(b, .{});
    const optimize = b.standardOptimizeOption(.{});

    const pdx = playdate.addCompileBundle(b, .{});

    for (pdex_targets) |target| {
        const pdex = playdate.addCompileExecutable(b, .{
            .root_source_file = .{ .path = "src/main.zig" },
            .target = target,
            .optimize = optimize,
        });
        pdx.addExecutable(pdex);
    }

    pdx.addAsset(.{ .path = "assets/pdxinfo" }, "pdxinfo");
    pdx.addAsset(.{ .path = "assets/Roobert-11-Mono-Condensed.fnt" }, "Roobert-11-Mono-Condensed.fnt");
    pdx.addAsset(.{ .path = "assets/Roobert-11-Mono-Condensed-table-8-16.png" }, "Roobert-11-Mono-Condensed-table-8-16.png");
    pdx.addAsset(.{ .path = "assets/sprites.png" }, "sprites.png");

    const installed_pdx = playdate.addInstallBundle(b, pdx, .prefix, "MyGame.pdx");
    b.getInstallStep().dependOn(&installed_pdx.installation.step);

    const run_simulator = playdate.addRunSimulator(b, installed_pdx);
    run_simulator.command.step.dependOn(b.getInstallStep());

    const run_step = b.step("run", "Run the game in the Playdate Simulator");
    run_step.dependOn(&run_simulator.command.step);
}

I feel this is very elegant and I would be very sad if the build system is taken in a direction that makes usages like these no longer possible.

[rohlem]

on the term "inversion of control" EDIT: which IMO isn't really the core of this discussion

To my understanding of the Wikipedia definition, "inversion of control" is a phrase used for patterns where the callee calls code provided by the caller.
For example, providing callbacks for a function to call once, repeatedly, or not at all.
(Since it's the callee's logic deciding this, whether and when the flow of control enters the callback becomes less obvious / more difficult to understand.)

I personally wouldn't call either of the examples posted here "inversion of control".
The imported functions seem to be called exactly once, and use the same std.Build-API that the parent build.zig would use in their place.
(They're cooperatively building a single build graph, but not calling each other back-and-forth to do so.)

---

EDIT:

I'm happy to work together on coming up with a satisfactory API that is based on dependencies exposing things rather than doing build logic

I guess this should be the main point to focus on.
The idea is probably that, to change the build logic of project A, a developer should have to edit only A's build.zig, and not code in one of its imports that does that work for A.

(Just for completeness, it also looks like the packages "playdate" and "microzig-build" respectively would be required unconditionally,
so with regards to the lazyDependency mechanism implemented in #18778 I don't see any motivation to have them marked lazy.)

[andrewrk]

Let's not get bogged down in the definition of "inversion of control". I'm sorry for using possibly overloaded language. We all know what is being discussed - only this exact pattern of @import a dependency in a build script, and how to make that work with lazy dependencies (if at all).

@castholm the best I can possibly do, while making import functionality support playdate being a lazy dependency would be something like this:

 const std = @import("std");
-const playdate = @import("playdate");
 
 pub fn build(b: *std.Build) void {
+    const playdate = b.importDependency("playdate") orelse return;
     const pdex_targets = playdate.standardExecutableTargetOptions(b, .{});
     const optimize = b.standardOptimizeOption(.{});
 
     const pdx = playdate.addCompileBundle(b, .{});

"playdate" would need to be a comptime known value, and the function could only be called from the package that had playdate listed in build.zig.zon.

Thanks for providing the examples, @MasterQ32 and @castholm. These use cases are certainly important and as far as I'm concerned any steps taken to change the build system must continue to address them satisfactorily.

Just to be crystal clear, the changes in the recently merged PR (#18778) did nothing to regress these use cases. Rather, it means that those packages being used with @import are, as it stands, ineligible to be used with the new lazy dependency feature.

[InKryption]

const playdate = b.importDependency("playdate") orelse return;

"playdate" would need to be a comptime known value, and the function could only be called from the package that had playdate listed in build.zig.zon.

What would the type of the playdate variable end up being here? As it stands it couldn't be of type type since it's taking a runtime parameter of type *std.Build.

[andrewrk]

It would be type type. It is indeed possible to take a runtime parameter and return a compile-time known value. Here is an example:

const std = @import("std");
const assert = std.debug.assert;

extern fn side_effect(*i32) i32;

inline fn example(runtime_known: *i32, comptime x: bool) i32 {
    if (x) {
        return 1234;
    } else {
        return side_effect(runtime_known);
    }
}

test "pass runtime param, receive comptime value" {
    const ptr = try std.testing.allocator.create(i32);
    defer std.testing.allocator.destroy(ptr);

    const result = example(ptr, true);
    comptime assert(result == 1234);
}

This test passes, however, if you flip the true to a false you get:

test.zig:19:28: error: unable to evaluate comptime expression
    comptime assert(result == 1234);
                    ~~~~~~~^~~~~~~
test.zig:19:21: note: operation is runtime due to this operand
    comptime assert(result == 1234);
                    ^~~~~~

However... when I tried this example below I was actually surprised:

const std = @import("std");

extern fn side_effect(*i32) void;

inline fn example(runtime_known: *i32, comptime x: bool) type {
    if (x) {
        return f32;
    } else {
        side_effect(runtime_known);
    }
}

test "pass runtime param, receive comptime value" {
    const ptr = try std.testing.allocator.create(i32);
    defer std.testing.allocator.destroy(ptr);

    const T = example(ptr, true);
    try std.testing.expect(T == f32);
}

test.zig:17:23: error: unable to resolve comptime value
    const T = example(ptr, true);
                      ^~~
test.zig:17:23: note: argument to function being called at comptime must be comptime-known
test.zig:5:58: note: expression is evaluated at comptime because the function returns a comptime-only type 'type'
inline fn example(runtime_known: *i32, comptime x: bool) type {
                                                         ^~~~
test.zig:5:58: note: types are not available at runtime
inline fn example(runtime_known: *i32, comptime x: bool) type {
                                                         ^~~~

The problem here is that the return type forces the function call to be comptime, when we actually just wanted it to be inline. I will open a bug report for this.

[andrewrk]

I believe this use case has been fully addressed by e204a6e.