Skip to content

communicate the error when a test fails in release mode #14972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Tracked by #14647
andrewrk opened this issue Mar 17, 2023 · 2 comments
Open
Tracked by #14647

communicate the error when a test fails in release mode #14972

andrewrk opened this issue Mar 17, 2023 · 2 comments
Labels
contributor friendly This issue is limited in scope and/or knowledge of Zig internals. enhancement Solving this issue will likely involve adding new logic or components to the codebase. zig build system std.Build, the build runner, `zig build` subcommand, package management
Milestone

Comments

@andrewrk
Copy link
Member

Extracted from #14647.

Running unit tests via the build system in release mode with a failure currently looks like this:

const std = @import("std");

test "oops" {
    try std.testing.expect(false);
}
andy@ark ~/t/abc [1]> zig build test -Doptimize=ReleaseFast
run test: error: 'test.oops' failed
run test: error: the following test command failed:
/home/andy/tmp/abc/zig-cache/o/a12abb147b3c725378efb29b200da6fa/test --listen=- 
Build Summary: 1/3 steps succeeded; 1 failed; 0/1 tests passed; 1 failed (disable with -fno-summary)
test transitive failure
└─ run test 0/1 passed, 1 failed
   └─ zig test ReleaseFast native success 6s MaxRSS:171M
error: the following build command failed with exit code 1:
/home/andy/tmp/abc/zig-cache/o/689850d0ce8aa0fb1a06ce5b6bf02f7b/build /home/andy/Downloads/zig/build-release/stage3/bin/zig /home/andy/tmp/abc /home/andy/tmp/abc/zig-cache /home/andy/.cache/zig test -Doptimize=ReleaseFast

There is one piece of information that is lost, which is the fact that the test failed with error.TestUnexpectedResult. This information could be communicated via the TestResult message over test protocol, or the test runner could simply print the error name to stderr before reporting a failure.

@andrewrk andrewrk added enhancement Solving this issue will likely involve adding new logic or components to the codebase. contributor friendly This issue is limited in scope and/or knowledge of Zig internals. zig build system std.Build, the build runner, `zig build` subcommand, package management labels Mar 17, 2023
@andrewrk andrewrk added this to the 0.11.0 milestone Mar 17, 2023
@brianwitte
Copy link

Hello, Sir! 😄

Trying to reproduce this with a fresh zig version compiled from master at 68c7261 without success.

I used an init-exe scaffold and replaced default test in src/main.zig with your oops test from above.

My output includes the failing test and error. My output also is generally different than yours which makes me think I a missing some config or flags that you have.

[bkz@knot bug-14972]$ uname -a
Linux knot 6.1.12-1-MANJARO #1 SMP PREEMPT_DYNAMIC Tue Feb 14 21:59:10 UTC 2023 x86_64 GNU/Linux

[bkz@knot bug-14972]$ zig version
0.11.0-dev.2150+68c7261e1

[bkz@knot bug-14972]$ zig build test -Doptimize=ReleaseFast
Test [1/1] test.oops... FAIL (TestUnexpectedResult)
0 passed; 0 skipped; 1 failed.
error: the following test command failed with exit code 1:
/home/bkz/bug-14972/zig-cache/o/f14aa9bb168cb06828e44a8d8bde174e/test
error: test...
error: The following command exited with error code 1:
/home/bkz/.zig/zig test /home/bkz/bug-14972/src/main.zig -OReleaseFast --cache-dir /home/bkz/bug-14972/zig-cache --global-cache-dir /home/bkz/.cache/zig --name test --enable-cache 
error: the following build command failed with exit code 1:
/home/bkz/bug-14972/zig-cache/o/a2c40919371c05801ade8a210be27864/build /home/bkz/.zig/zig /home/bkz/bug-14972 /home/bkz/bug-14972/zig-cache /home/bkz/.cache/zig test -Doptimize=ReleaseFast

[bkz@knot bug-14972]$ cat build.zig 
const std = @import("std");

// Although this function looks imperative, note that its job is to
// declaratively construct a build graph that will be executed by an external
// runner.
pub fn build(b: *std.Build) void {
    // Standard target options allows the person running `zig build` to choose
    // what target to build for. Here we do not override the defaults, which
    // means any target is allowed, and the default is native. Other options
    // for restricting supported target set are available.
    const target = b.standardTargetOptions(.{});

    // Standard optimization options allow the person running `zig build` to select
    // between Debug, ReleaseSafe, ReleaseFast, and ReleaseSmall. Here we do not
    // set a preferred release mode, allowing the user to decide how to optimize.
    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "bug-14792",
        // In this case the main source file is merely a path, however, in more
        // complicated build scripts, this could be a generated file.
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    // This declares intent for the executable to be installed into the
    // standard location when the user invokes the "install" step (the default
    // step when running `zig build`).
    exe.install();

    // This *creates* a RunStep in the build graph, to be executed when another
    // step is evaluated that depends on it. The next line below will establish
    // such a dependency.
    const run_cmd = exe.run();

    // By making the run step depend on the install step, it will be run from the
    // installation directory rather than directly from within the cache directory.
    // This is not necessary, however, if the application depends on other installed
    // files, this ensures they will be present and in the expected location.
    run_cmd.step.dependOn(b.getInstallStep());

    // This allows the user to pass arguments to the application in the build
    // command itself, like this: `zig build run -- arg1 arg2 etc`
    if (b.args) |args| {
        run_cmd.addArgs(args);
    }

    // This creates a build step. It will be visible in the `zig build --help` menu,
    // and can be selected like this: `zig build run`
    // This will evaluate the `run` step rather than the default, which is "install".
    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);

    // Creates a step for unit testing.
    const exe_tests = b.addTest(.{
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    // Similar to creating the run step earlier, this exposes a `test` step to
    // the `zig build --help` menu, providing a way for the user to request
    // running the unit tests.
    const test_step = b.step("test", "Run unit tests");
    test_step.dependOn(&exe_tests.step);
}

[bkz@knot bug-14972]$ cat src/main.zig 
const std = @import("std");

pub fn main() !void {
    // Prints to stderr (it's a shortcut based on `std.io.getStdErr()`)
    std.debug.print("All your {s} are belong to us.\n", .{"codebase"});

    // stdout is for the actual output of your application, for example if you
    // are implementing gzip, then only the compressed bytes should be sent to
    // stdout, not any debugging messages.
    const stdout_file = std.io.getStdOut().writer();
    var bw = std.io.bufferedWriter(stdout_file);
    const stdout = bw.writer();

    try stdout.print("Run `zig build test` to run the tests.\n", .{});

    try bw.flush(); // don't forget to flush!
}

test "oops" {
    try std.testing.expect(false);
}

Perhaps I need to make some adjustments to get a proper repro? Let me know what else I can do and in the mean time, I will try a few more things.

@andrewrk
Copy link
Member Author

If I had to guess I'd say that you are using an old lib/ directory. If that is the problem, then you either need to delete the lib directory and start using the "no lib" option, or you need to run the install step to copy the lib folder from the zig source tree to your installation prefix where your zig binary lives.

@andrewrk andrewrk modified the milestones: 0.11.0, 0.12.0 Jul 20, 2023
@andrewrk andrewrk modified the milestones: 0.13.0, 0.12.0 Aug 5, 2023
@andrewrk andrewrk modified the milestones: 0.12.0, 0.13.0 Mar 21, 2024
@andrewrk andrewrk modified the milestones: 0.14.0, 0.15.0 Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contributor friendly This issue is limited in scope and/or knowledge of Zig internals. enhancement Solving this issue will likely involve adding new logic or components to the codebase. zig build system std.Build, the build runner, `zig build` subcommand, package management
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants