-
-
Notifications
You must be signed in to change notification settings - Fork 2.8k
integrate fuzz testing into the build system #20773
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
726cec6
to
6c68370
Compare
for the -fno-fuzz case. The other case will take more work in libfuzzer.
For now this returns a dummy fuzz input.
This flag makes the build runner rebuild unit tests after the pipeline finishes, if it finds any unit tests. I did not make this integrate with file system watching yet. The test runner is updated to detect which tests are fuzz tests. Run step is updated to track which test indexes are fuzz tests.
This way they can be smoke tested.
The flag makes compiler_rt and libfuzzer be in debug mode. Also: * fuzzer: override debug logs and disable debug logs for frequently called functions * std.Build.Fuzz: fix bug of rerunning the old unit test binary * report errors from rebuilding the unit tests better * link.Elf: additionally add tsan lib and fuzzer lib to the hash
just some experimentation. I didn't expect this to be effective so quickly but it already can find a comparison made with mem.eql
seems to provide better scoring
This was referenced Jul 26, 2024
Closed
Open
In case these get added I want to propose the name |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
fuzzing
release notes
This PR should be mentioned in the release notes.
zig build system
std.Build, the build runner, `zig build` subcommand, package management
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a
--fuzz
CLI option to the build runner. When this is used it rebuilds any unit test binaries which contained at least one fuzz test with-ffuzz
and then tells it to start fuzzing, which does in-process fuzzing.Adds
std.testing.fuzzInput
, which is how unit tests mark themselves as fuzz tests.This contains only a rudimentary implementation of fuzzer logic, really just some early, early experiments, but already it makes this test case fail in 65 milliseconds on my machine:
asciinema demo
Closes #20702.
Follow-Up Tasks
std.testing.fuzzInput
: introduce a corpus option #20814std.testing.fuzzInput
: ability to choose whether timing out is considered a fail or a pass #20815std.testing.fuzzInput
: introduce a length range option #20816zig build --fuzz
instances running on different machines--fuzz
with--watch