package manager: Transitive dependencies have incorrect hash #17282
Assignees
Labels
bug
Observed behavior contradicts documented or intended behavior
os-linux
upstream
An issue with a third party project that Zig uses.
zig build system
std.Build, the build runner, `zig build` subcommand, package management
Milestone
Comments
Southporter
added
the
bug
|
Are you on btrfs? If so then this is probably caused by #17095. |
|
Yes, I'm on btrfs. One thing to note, this only seems to be an issue on x86_64. I have a Kinoite vm on aarch64 also on btrfs, but does not have the issue. I'll look at that other thread. |
|
It does indeed seem to be btrfs related. I created a ext4 partition and moved the zig cache to that partition and it is pulling down and calculating the package hashes correctly. Current workaround: |
andrewrk
added
zig build system
|
Closing as duplicate of #17095. Unfortunately, we have nothing to do but wait until users move on from the affected kernel versions. Short summary of the problem:
Recommend for btrfs users to stick with < 6.4.12 until a kernel version is released that has the fix. |
|
I have an idea to make this slightly better for Zig users. We can have an opt-in environment variable |
andrewrk
added a commit
that referenced
this issue
Oct 4, 2023
Organize everything around a Fetch task which does a bunch of stuff in a worker thread without touching any shared state, and then queues up Fetch tasks for its dependencies. This isn't the theoretical optimal package fetching performance because CPU cores don't necessarily map 1:1 with I/O tasks, and each fetch task contains a mixture of computations and I/O. However, it is expected for this to significantly outperform master branch, which fetches everything recursively with only one thread. The logic is now a lot more linear and easy to follow. Everything that is embarassingly parallel is done on the thread pool, and then after everything is fetched, the worker threads are joined and the main thread does the finishing touches of stitching together the dependencies.zig import files. There is only one tiny little critical section and it does not even have any error handling in it. This also lays the groundwork for #14281 because in system mode, all this fetching logic will be skipped, but the "finishing touches" mentioned above still need to be done. With this branch, that logic is separated out and no longer recursively tangled with fetching stuff. Additionally, this branch: * Implements inclusion directives in `build.zig.zon` for deciding which files belong the package (#14311). * Adds basic documentation for `build.zig.zon` files. * Adds support for fetching dependencies with the `file://` protocol scheme (#17364). * Adds a workaround for a Linux/btrfs file system bug (#17282). This commit is a work-in-progress. Still todo: 1. Hook up the CLI to the new system. 2. Restore the module table creation logic after all the fetching is done. 3. Fix compilation errors, get the tests passing, and regression test against real world projects.
andrewrk
added a commit
that referenced
this issue
Oct 7, 2023
Organize everything around a Fetch task which does a bunch of stuff in a worker thread without touching any shared state, and then queues up Fetch tasks for its dependencies. This isn't the theoretical optimal package fetching performance because CPU cores don't necessarily map 1:1 with I/O tasks, and each fetch task contains a mixture of computations and I/O. However, it is expected for this to significantly outperform master branch, which fetches everything recursively with only one thread. The logic is now a lot more linear and easy to follow. Everything that is embarassingly parallel is done on the thread pool, and then after everything is fetched, the worker threads are joined and the main thread does the finishing touches of stitching together the dependencies.zig import files. There is only one tiny little critical section and it does not even have any error handling in it. This also lays the groundwork for #14281 because in system mode, all this fetching logic will be skipped, but the "finishing touches" mentioned above still need to be done. With this branch, that logic is separated out and no longer recursively tangled with fetching stuff. Additionally, this branch: * Implements inclusion directives in `build.zig.zon` for deciding which files belong the package (#14311). * Adds basic documentation for `build.zig.zon` files. * Adds support for fetching dependencies with the `file://` protocol scheme (#17364). * Adds a workaround for a Linux/btrfs file system bug (#17282). This commit is a work-in-progress. Still todo: 1. Hook up the CLI to the new system. 2. Restore the module table creation logic after all the fetching is done. 3. Fix compilation errors, get the tests passing, and regression test against real world projects.
andrewrk
added a commit
that referenced
this issue
Oct 8, 2023
Organize everything around a Fetch task which does a bunch of stuff in a worker thread without touching any shared state, and then queues up Fetch tasks for its dependencies. This isn't the theoretical optimal package fetching performance because CPU cores don't necessarily map 1:1 with I/O tasks, and each fetch task contains a mixture of computations and I/O. However, it is expected for this to significantly outperform master branch, which fetches everything recursively with only one thread. The logic is now a lot more linear and easy to follow. Everything that is embarassingly parallel is done on the thread pool, and then after everything is fetched, the worker threads are joined and the main thread does the finishing touches of stitching together the dependencies.zig import files. There is only one tiny little critical section and it does not even have any error handling in it. This also lays the groundwork for #14281 because in system mode, all this fetching logic will be skipped, but the "finishing touches" mentioned above still need to be done. With this branch, that logic is separated out and no longer recursively tangled with fetching stuff. Additionally, this branch: * Implements inclusion directives in `build.zig.zon` for deciding which files belong the package (#14311). * Adds basic documentation for `build.zig.zon` files. * Adds support for fetching dependencies with the `file://` protocol scheme (#17364). * Adds a workaround for a Linux/btrfs file system bug (#17282). This commit is a work-in-progress. Still todo: 1. Hook up the CLI to the new system. 2. Restore the module table creation logic after all the fetching is done. 3. Fix compilation errors, get the tests passing, and regression test against real world projects.
|
Workaround landed in f7bc55c. Please export the environment variable |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Zig Version
0.11.0 and 0.12.0-dev.575+fb6fff256
Steps to Reproduce and Observed Behavior
Create an new program with
zig init-exeAdd one of the attached
build.zig.zonfiles.Run
zig build run.The hash of the packages dependencies will be incorrect. Seems to happen for any package that has dependencies.
This only seems to happen on my linux machine. Details below. For MacOS this works fine. I will try it on some other machines if I can get the time to spin up a few vms, or try a live usb.
Machine Information:
Relevant details from cat of /etc/os-release
NAME="Fedora Linux"VERSION="38.20230925.0 (Kinoite)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38.20230925.0 (Kinoite)"
VARIANT="Kinoite"
VARIANT_ID=kinoite
OSTREE_VERSION='38.20230925.0'
Example build.zig.zon files
build.zig.zon with treez dependency
build.zig.zon with mach-freetype dependency
Expected Behavior
zig build runshould pull dependencies and hashes should be calculated correctly.The text was updated successfully, but these errors were encountered: