Skip to content

ability to depend on pure zig modules without running any build.zig logic #14282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Tracked by #14265
andrewrk opened this issue Jan 12, 2023 · 10 comments
Closed
Tracked by #14265
Labels
enhancement Solving this issue will likely involve adding new logic or components to the codebase. proposal This issue suggests modifications. If it also has the "accepted" label then it is planned. zig build system std.Build, the build runner, `zig build` subcommand, package management
Milestone

Comments

@andrewrk
Copy link
Member

andrewrk commented Jan 12, 2023

Extracted from #14265.

Terminology clarification: #14307

For many zig modules, advanced build system features are not needed, such as generating source files, compiling C code, or accepting build options. In such cases, build.zig is overkill. All that is really needed is the .zig source files to be available to the project which has the dependency on this one.

For this I propose the package to be organized like this:

  • build.zig - may or may not exist. Perhaps the package still has a test harness to be run, or has development dependencies.
  • build.zig.zon - may or may not exist.

Meanwhile the package depending on it will do something like this:

build.zig.zon

.{
    .name = "clap",
    .url = "...",
    .hash = "...",
}

build.zig

// "clap" here references the dependency.name field of the build.zig.ini file.
const exe = b.addExecutable("app", "src/main.zig");
exe.addModuleFromDependencyPath(
    // name to be used by @import
    "clap",
    // module root directory, relative to clap dependency build root
    ".",
    // module root source file, relative to module root directory
    "clap.zig",
);

Note that this allows a project to add a dependency based on any arbitrary directory of files without the dependee being aware of it being used that way.

When a module is added this way, no build.zig logic is executed. Note that this could be problematic, such as in the case of a dependency introducing a new generated file as part of their package, or adding build options. To future-proof against this situation, instead of addModuleFromDependencyPath, build scripts should use b.dependency() which will execute the dependency's build logic.

See #14278 for depending on zig packages while respecting build.zig logic.

See #14286 for running build.zig logic in a sandbox.

@andrewrk andrewrk added enhancement Solving this issue will likely involve adding new logic or components to the codebase. zig build system std.Build, the build runner, `zig build` subcommand, package management labels Jan 12, 2023
@andrewrk andrewrk added this to the 0.11.0 milestone Jan 12, 2023
@deflock
Copy link

deflock commented Jan 13, 2023

The filename, build.zig.ini is intended to imply that it is an appendage to build.zig because that is exactly what it is. The real, actual file that signifies a zig package is a build.zig, and the existence of this extra file is bonus - it is for the case of declarative information that we want to expose without requiring execution of zig code.

Is this still relevant with this change?

@nektro
Copy link
Contributor

nektro commented Jan 13, 2023

since entry point and transitive dependencies are likely to be declared by the source dependency itself, imo it would make more sense for build.zig.toml to be required and to make addDependencyPackagePath() accept a name and a path to a build.zig.toml rather than a zig source file

@ikskuh
Copy link
Contributor

ikskuh commented Jan 13, 2023

since entry point and transitive dependencies are likely to be declared by the source dependency itself, imo it would make more sense for build.zig.toml to be required and to make addDependencyPackagePath() accept a name and a path to a build.zig.toml rather than a zig source file

An alternative to that is to use the usual build.zig.${ext} approach and encode the exported packages inside that file:

[project]
name="tiny-package-bundle-3"

[[package]]
name="leftpad"
root="src/leftpad.zig"

[[package]]
name="rightpad"
root="src/rightpad.zig"
dependency=["stringhandler.stralloc"] # imports the package 'stralloc' from the dependency 'stringhandler'

[[dependency]]
name="stringhandler"
url="https://ohno.example.com"

So we can just use the dependency in the main build.zig:

pub fn build(b: *std.build.Builder) void {
   …
   exe.addPackage("tiny-leftpad", b.dependency("tiny-package-bundle-3").package("rightpad"));
   …
}

This will allow zig-only projects to import other projects, without having to use build.zig logic for the project. Also a project can export more than one package.

Use case for this:
Zig Embedded Group can provide family support bundles, for example "AVR" as a project, that exports all the AVR family microcontroller support packages (which would be a huge load of small packages like atmega328p, atmega32, atmega8515, …)

@andrewrk andrewrk changed the title ability to depend on pure zig packages without running any build.zig logic ability to depend on pure zig modules without running any build.zig logic Jan 13, 2023
@david-vanderson
Copy link
Contributor

Can we do this recursively? So an app (with build logic) that uses a pure zig dependency A that uses a pure zig dependency B, and neither A nor B have build logic?

When developing library A, how do I import stuff from B without a build.zig?

@andrewrk
Copy link
Member Author

That's a great point. It demonstrates that this idea only works if it can be encoded via purely the declarative file.

@leroycep
Copy link
Contributor

It would be nice to just get access to the raw files. There already projects that have multiple Zig packages in a repository, and you just pass it a path to the file. What if the path to package was just exposed?

// "clap" here references the dependency.name field of the build.zig.ini file.
const exe = b.addExecutable("app", "src/main.zig");
exe.addPackage(.{
    .name = "clap",
    .source = .{ .path = b.pathJoin(&.{ b.packagePath("clap"), "clap.zig" })  },
});

Failing that, there is the @src().file pattern that could be used in conjunction with #14279 allow the build.zig script to create the Pkg for module's user. However, that defeats the purpose of this proposal and it feels like it should work more along the lines of adding a c dependency.

I personally would like to use something like b.packagePath("clap") to depend on a binary release. For example I depend on ZWO ASI's SDK, which only contains binary .a and .so files. While I could have some other system of getting the library files (like say, checking it into a git repository), it would also be nice to specify that in the build.zig.ini, and then add the include/library path using b.packagePath("asicamera").

(Actually, in this case I probably wouldn't link there because they don't version the download link. But I don't think this is the only case where a project depends on a binary file downloaded from somewhere else.)

@andrewrk
Copy link
Member Author

I'm changing this to a proposal. I'm starting to get the impression that this may be a terrible mistake.

@andrewrk andrewrk modified the milestones: 0.11.0, 0.12.0 Jan 28, 2023
@andrewrk andrewrk added the proposal This issue suggests modifications. If it also has the "accepted" label then it is planned. label Jan 28, 2023
@leroycep
Copy link
Contributor

Could you say what makes you think that? I can see fetching arbitrary archive files as perhaps distasteful; but without a centralized server there is only so much that can be done to prevent it. Especially if #14294 is implemented, any plugin could preprocess the file to make it work with the package format.

Anyway, I just want to know if there's a reason to specifically disallow treating fetched packages as another folder you have access to.

@david-vanderson
Copy link
Contributor

I think the option to fetch arbitrary archives is still on the table, but should be considered separate from this issue. See #14488

For pure zig dependencies, I'm starting to agree with @andrewrk. If there's no build.zig, then somehow putting things into build.zig.zon would have to magically alter what you can import in zig files. Then for a given zig file the compiler would have to find the "right" build.zig.zon.

The alternative is mandating a build.zig file to be able to use any dependencies, which seems straightforward and likely in any case.

@andrewrk
Copy link
Member Author

I am rejecting this proposal to avoid the problem of a package which needs to introduce build logic, but is unable to due to having dependents which would be broken by such a change. Similarly, a project may wish to move source files around, and should not need to worry about a dependent which has hard-coded the file path of the source files.

This decision could be reversed later; however I think the best approach at least for now is to reject this proposal.

@andrewrk andrewrk closed this as not planned Won't fix, can't repro, duplicate, stale Apr 19, 2023
@andrewrk andrewrk modified the milestones: 0.12.0, 0.11.0 Apr 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Solving this issue will likely involve adding new logic or components to the codebase. proposal This issue suggests modifications. If it also has the "accepted" label then it is planned. zig build system std.Build, the build runner, `zig build` subcommand, package management
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

6 participants