support fetch plugins to download dependencies from less common protocols

[andrewrk]

Extracted from #14265.

Zig will have built-in support for fetching dependencies from some protocols, such as http, https, gzip, tar, and some others. But there will always be new or exotic protocols. One such example would be ipfs. Not really ubiquitous enough to support directly, but could be interesting for some people to use.

For this I propose a "fetch plugin" system.

Fetch plugins would be specified in build.zig.zon like this:

.{
    // ...
    .fetch_plugins = .{
        .ipfs = .{
            .url = "git+ssh://[email protected]/foo.tar.gz#6f987aba83414319c3afba57a9f49a71f6e13c8e",
            .hash = "sha256=c9b30cffc40999d2c078ff350cbcee642970a224fe123c756d0892f876cf1aae",
        },
        .@"git+ssh" = .{
            .url = "http://example.com/bar.tar.gz",
            .hash = "sha256=9ba4f49895b174a3f918d489238acbc146bd393575062b2e3be33488b688e36f",
        },
    },
    // ...
}

The url fields within fetch_plugins must use a built-in URI scheme or a different fetch plugin from the same manifest.

Implementing this proposal will lift out this code from the zig compiler:

zig/src/main.zig

Lines 4080 to 4126 in 7cb2f92

	if (!build_options.omit_pkg_fetching_code) {
	var http_client: std.http.Client = .{ .allocator = gpa };
	defer http_client.deinit();
	try http_client.rescanRootCertificates();
	// Here we provide an import to the build runner that allows using reflection to find
	// all of the dependencies. Without this, there would be no way to use `@import` to
	// access dependencies by name, since `@import` requires string literals.
	var dependencies_source = std.ArrayList(u8).init(gpa);
	defer dependencies_source.deinit();
	try dependencies_source.appendSlice("pub const imports = struct {\n");
	// This will go into the same package. It contains the file system paths
	// to all the build.zig files.
	var build_roots_source = std.ArrayList(u8).init(gpa);
	defer build_roots_source.deinit();
	// Here we borrow main package's table and will replace it with a fresh
	// one after this process completes.
	main_pkg.fetchAndAddDependencies(
	&thread_pool,
	&http_client,
	build_directory,
	global_cache_directory,
	local_cache_directory,
	&dependencies_source,
	&build_roots_source,
	"",
	) catch |err| switch (err) {
	error.PackageFetchFailed => process.exit(1),
	else => |e| return e,
	};
	try dependencies_source.appendSlice("};\npub const build_root = struct {\n");
	try dependencies_source.appendSlice(build_roots_source.items);
	try dependencies_source.appendSlice("};\n");
	const deps_pkg = try Package.createFilePkg(
	gpa,
	local_cache_directory,
	"dependencies.zig",
	dependencies_source.items,
	);
	mem.swap(Package.Table, &main_pkg.table, &deps_pkg.table);
	try main_pkg.addAndAdopt(gpa, "@dependencies", deps_pkg);
	}

...and move it to a new file lib/fetch_runner.zig. This is similar to build_runner.zig and test_runner.zig and has the responsibility to fetch the full dependency tree. It will deal with fetch plugins by fetching them, and then rebuilding the fetch runner itself, multiple times if necessary, until everything is fetched.

Once everything is fetched, build_runner.zig proceeds as usual. Note that in the case when everything is already fetched, the fetch_runner will not be executed because the open() syscalls on the first-level dependencies based on their hashes all succeeded.

[AdamGoertz]

How would this fit into the framework proposed in #14286? It sounds like the goal in that issue is to make it safer to run un-audited build.zig files from dependencies by sandboxing them with WASM, but this proposal would also allow arbitrary code execution by downloading and running fetch plugins, similarly to running someone else's build.zig.

[ArtemKolichenkov]

As a user of IPFS once in a blue moon, here's my 2 cents.

If you found a package that for some reason exists only on IPFS, you can

• use a IPFS gateway (local or remote), then feed its url to zig
• just download tar manually if it's a one-time occasion

This is applicable to practically any other uncommon protocols and are generally straightforward to implement. It is also guaranteed to work with Zig (it would just see it as normal http/tar).
For those who frequently use exotic protocols, they likely already have the expertise to employ standard proxy methods. If a proxy for a specific protocol isn't available, developing a general-purpose proxy could be more beneficial than creating a plugin specifically for the Zig package manager.

Even without considering security implications - this just seems like a feature creep.

Given that the package manager is still in its early development phase, focusing on foundational stability and functionality might be more prudent. I wouldn't expect this kind of feature to be available even in Zig 1.0, let alone 0.13.0

[daurnimator]

This is applicable to practically any other uncommon protocols and are generally straightforward to implement. It is also guaranteed to work with Zig (it would just see it as normal http/tar). For those who frequently use exotic protocols, they likely already have the expertise to employ standard proxy methods. If a proxy for a specific protocol isn't available, developing a general-purpose proxy could be more beneficial than creating a plugin specifically for the Zig package manager.

Even without considering security implications - this just seems like a feature creep.

Agreed, see also my comment at #14298 (comment)

[KilianHanich]

So, I decided that I try to tackle this over the last few days, so, here is my rather basic proposal. It may have some flaws which I overlooked, but I think it's fine for what one can get without an implementation. It makes it quite easy to e.g. implement Git over SSH (for a few reason which I mentioned in that issue #14295).

As for security considerations. The way I have formulated this one could end up restricting it in a way to make it run inside of a sandbox (e.g only access to the network and the target directory and maybe some directory for temporary files), but since for example Git over SSH users often use their SSH keys to connect to the server, I am not sure how practical that would end up being.

Anyway, here we go:

A fetch plugin is a directory with the following contents:

• a file called fetch-plugin.zon; this file is a struct with the following fields:
  • .projectname: This is the name of the plugin
  • .protocols: a list of all supported protocols of this plugin
  • .kind: an enum of two possible values (.Source and .Executable) which influence the other expected contents of the plugin
• If it is an .Executable plugin, the directory contains an executable file (or a symlink to one) which does the fetching. How the interface works is described later. The name of the executable must the Projectname plus possible OS-specific file extensions. This is mostly meant for Fetch Plugins which are scripts (like a bash file or a batch file) or precompiled.
• If it is a .Source plugin, the directory contains a Zig project whose build.zig file supports zig build run -- with the arguments coming afterwards.
• A Fetch Plugin is not allowed to depend on Fetch Plugins to work. Be it directly or indirectly.

The plugin is supposed to behave according to the following interface:

• The exit code must be 0 if it is successful.
• The first parameter is where the target should end up in a useful state (like decompressed).
• The second parameter is the URL of the target to be fetched.
• It is not allowed to make use of STDIN, STDOUT and STDERR.
• It may make use of the Zig Progress Protocol Specification for progress communication.

Here an quick'n'dirty example:

Directory with the files fetch-plugin.zon, git-plugin.sh and git-plugin.bat.

• content of fetch-plugin.zon:

    .{
        .projectname = "git-plugin",
        .protocols = .{
            "git+http",
            "git+https",
            "git+ssh",
        },
        .kind = .Executable,
    }
  

• content of git-plugin.sh:

    #!/bin/sh
    git clone --depth=1 "$2" "$1" >/dev/null 2>/dev/null
    exit $?
  

• content of git-plugin.bat

    @echo off
    git clone --depth=1 %2 %1 > nul 2> nul
    exit %errorlevel%