Better support for fuzzing Rust, remaining - code coverage; MSan; LSan in ASan

[Dor1s]

Tracking bug for ideas / suggestions

[jonathanmetzman]

Fixing this inconsistency/getting the normal thing working on jenkins: #3298

[jfoote]

IIUC rust-lang/rust#34701 could be a precursor to adding support for something like clang source-based coverage for Rust targets (related issue: #3468).

[catenacyber]

Support MSAN when mixing C and Rust cf
#2145 (comment)

[inferno-chromium]

Didnt realise sanitizers were disabled for Rust projects, this definitely seems bad.

oss-fuzz/projects/mp4parse-rust/build.sh

Line 24 in 44f4b89

# Because Rust does not support sanitizers via CFLAGS/CXXFLAGS, the environment

[inferno-chromium]

#3830 simplifies rust workflow, adds missing sanitizer in many projects. migrated most in #3840, libra is remaining.

[inferno-chromium]

@catenacyber - MSan should work after #3830, maybe you want to try with some projects ? If you can also find what things are missing, that would be much appreciated.

[catenacyber]

Interesting, now I get a build failure ( bug found by MSAN ?) :

RUSTUP_HOME=/rust/rustup \
	CARGO_HOME="/rust" \
	CARGO_TARGET_DIR="/src/suricata/rust/target" \
	/rust/bin/cargo build --release \
		--features " " 
   Compiling autocfg v1.0.0
   Compiling cfg-if v0.1.9
   Compiling semver-parser v0.7.0
   Compiling libc v0.2.70
   Compiling arrayvec v0.4.12
   Compiling bitflags v1.2.1
   Compiling ryu v1.0.4
   Compiling memchr v2.3.3
   Compiling version_check v0.9.1
   Compiling nodrop v0.1.14
   Compiling getrandom v0.1.14
   Compiling static_assertions v0.3.4
   Compiling ppv-lite86 v0.2.8
   Compiling siphasher v0.3.3
   Compiling proc-macro2 v0.4.30
   Compiling unicode-xid v0.1.0
   Compiling syn v0.15.44
   Compiling build_const v0.2.1
   Compiling byteorder v1.3.4
   Compiling num-derive v0.2.5
   Compiling base64 v0.11.0
   Compiling widestring v0.4.0
   Compiling semver v0.9.0
   Compiling num-traits v0.2.11
   Compiling num-integer v0.1.42
   Compiling num-bigint v0.2.6
   Compiling num-rational v0.2.4
   Compiling num-iter v0.1.40
   Compiling num-complex v0.2.4
error: failed to run custom build command for `libc v0.2.70`

Caused by:
  process didn't exit successfully: `/src/suricata/rust/target/release/build/libc-774c2aec654d22a9/build-script-build` (exit code: 77)
--- stderr
Uninitialized bytes in __interceptor_memchr at offset 0 inside [0x701000000000, 4)
==19952==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55f8b392f2ee in std::sys::unix::memchr::memchr::hd921ac10d14dc82d /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/sys/unix/memchr.rs:6:9
    #1 0x55f8b392f2ee in std::memchr::memchr::h75af9c209e288688 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/memchr.rs:25:5
    #2 0x55f8b392f2ee in std::ffi::c_str::CString::_new::hd3fc15f3d02971b6 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/ffi/c_str.rs:355:15
    #3 0x55f8b392f2ee in std::ffi::c_str::CString::new::h2da21344d9607d80 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/ffi/c_str.rs:351:9
    #4 0x55f8b392f2ee in std::thread::Thread::new::_$u7b$$u7b$closure$u7d$$u7d$::h957656e8072d73d0 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/thread/mod.rs:1143:26
    #5 0x55f8b392f2ee in core::option::Option$LT$T$GT$::map::h5c5d6edf546fcd7f /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libcore/option.rs:458:29
    #6 0x55f8b392f2ee in std::thread::Thread::new::h5dfa556c4c696141 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/thread/mod.rs:1143:13
    #7 0x55f8b3938194 in std::rt::lang_start_internal::h9a98dc8e2c22711e /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/rt.rs:44:22
    #8 0x55f8b3927beb in main (/src/suricata/rust/target/release/build/libc-774c2aec654d22a9/build-script-build+0x6fbeb)
    #9 0x7f0ac815082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #10 0x55f8b38c7478 in _start (/src/suricata/rust/target/release/build/libc-774c2aec654d22a9/build-script-build+0xf478)

SUMMARY: MemorySanitizer: use-of-uninitialized-value /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/sys/unix/memchr.rs:6:9 in std::sys::unix::memchr::memchr::hd921ac10d14dc82d

Adding export MSAN_OPTIONS="halt_on_error=0:exitcode=0:report_umrs=0" gives then

   Compiling rand_core v0.5.1
error: couldn't read /src/suricata/rust/target/release/build/crc-43db1b02c9ee945c/out/crc16_constants.rs: No such file or directory (os error 2)
 --> /rust/registry/src/github.com-1ecc6299db9ec823/crc-1.8.1/src/crc16.rs:8:1
  |
8 | include!(concat!(env!("OUT_DIR"), "/crc16_constants.rs"));
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

error: could not compile `crc`.

[alexcrichton]

Ah the problem with that @catenacyber is that the usage of RUSTFLAGS is also affecting build scripts, which means sanitizers are running as part of the build (unfortunately). That can be fixed with a --target flag, however.

[catenacyber]

Thanks @alexcrichton
I am not sure I understand what you mean.
Where should I add the --target flag ? to the cargo build command ? With which value ?

More info, when trying to build with ASAN, I get error: /tmp/cargo-installDd2VEJ/release/deps/libserde_derive-e8fab3128f048517.so: undefined symbol: __asan_stack_free_7

[alexcrichton]

Yes you can add --target to cargo build, and you'll probably want --target x86_64-unknown-linux-gnu. That second error you get is because procedural macros are trying to get built with sanitizers, which doesn't work.

[catenacyber]

Thanks @alexcrichton
Using export CARGO_BUILD_TARGET="x86_64-unknown-linux-gnu" made it work.

Then @inferno-chromium it seems I am back to the previous problem of the fuzz target mixing C and Rust :
I get

==11==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0xd71b43 in core::result::Result$LT$T$C$E$GT$::unwrap::h94893129de2c63ee /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libcore/result.rs:1004:13
    #1 0xd71b43 in rs_dns_udp_register_parser /src/suricata/rust/src/dns/dns.rs:993:24

with the line being let default_port = std::ffi::CString::new("[53]").unwrap();

[inferno-chromium]

@catenacyber - i have fixed this now in #3859 , base build images should be rebuilt in one hour, then you need to infra/helper.py pull_images and retry.

[inferno-chromium]

Ok for MSan, check_build fails (which actually runs the binaries). @alexcrichton , can you please check below. Many projects use cargo fuzz build, is there a difference.

python infra/helper.py pull_images
python infra/helper.py build_fuzzers --sanitizer memory mp4parse-rust
python infra/helper.py check_build --sanitizer memory mp4parse-rust

Running: docker run --rm --privileged -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=memory -e ARCHITECTURE=x86_64 -v /build/oss-fuzz/build/out/mp4parse-rust:/out -t gcr.io/oss-fuzz-base/base-runner test_all
INFO: performing bad build checks for /tmp/not-out/avif.
INFO: performing bad build checks for /tmp/not-out/mp4.
Broken fuzz targets (2):
avif:
BAD BUILD: /tmp/not-out/avif seems to have either startup crash or exit:
INFO: Seed: 1337
INFO: Loaded 1 modules   (25295 inline 8-bit counters): 25295 [0x55eb5fd167a8, 0x55eb5fd1ca77), 
INFO: Loaded 1 PC tables (25295 PCs): 25295 [0x55eb5fd1ca78,0x55eb5fd7f768), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
==78==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55eb5f4fa87f in rust_fuzzer_test_input /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20
    #1 0x55eb5f71173f in __rust_try (/tmp/not-out/avif+0x36873f)
    #2 0x55eb5f711324 in std::panicking::try::hfc14f47abd791d38 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panicking.rs:274:15
    #3 0x55eb5f711324 in std::panic::catch_unwind::h3c37ac4d75f6cfa4 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panic.rs:394:14
    #4 0x55eb5f711324 in LLVMFuzzerTestOneInput /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:25:22
    #5 0x55eb5f786a4d in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:556:15
    #6 0x55eb5f78af58 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:743:3
    #7 0x55eb5f78c71e in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:794:3
    #8 0x55eb5f726f28 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:826:6
    #9 0x55eb5f70f950 in main /src/libfuzzer/FuzzerMain.cpp:19:10
    #10 0x7fcf97fc382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #11 0x55eb5f473398 in _start (/tmp/not-out/avif+0xca398)

DEDUP_TOKEN: rust_fuzzer_test_input--__rust_try--std::panicking::try::hfc14f47abd791d38
SUMMARY: MemorySanitizer: use-of-uninitialized-value /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20 in rust_fuzzer_test_input
  ORIGIN: invalid (0). Might be a bug in MemorySanitizer origin tracking.
    This could still be a bug in your code, too!
Unique heap origins: 35
Stack depot allocated bytes: 2376
Unique origin histories: 10
History depot allocated bytes: 240
Exiting
MS: 0 ; base unit: 0000000000000000000000000000000000000000


artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64: 
mp4:
BAD BUILD: /tmp/not-out/mp4 seems to have either startup crash or exit:
INFO: Seed: 1337
INFO: Loaded 1 modules   (25344 inline 8-bit counters): 25344 [0x556b13d2f7d8, 0x556b13d35ad8), 
INFO: Loaded 1 PC tables (25344 PCs): 25344 [0x556b13d35ad8,0x556b13d98ad8), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
==82==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x556b135130dc in rust_fuzzer_test_input /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20
    #1 0x556b13729f9f in __rust_try (/tmp/not-out/mp4+0x369f9f)
    #2 0x556b13729b84 in std::panicking::try::hfc14f47abd791d38 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panicking.rs:274:15
    #3 0x556b13729b84 in std::panic::catch_unwind::h3c37ac4d75f6cfa4 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panic.rs:394:14
    #4 0x556b13729b84 in LLVMFuzzerTestOneInput /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:25:22
    #5 0x556b1379f2ad in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:556:15
    #6 0x556b137a37b8 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:743:3
    #7 0x556b137a4f7e in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:794:3
    #8 0x556b1373f788 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:826:6
    #9 0x556b137281b0 in main /src/libfuzzer/FuzzerMain.cpp:19:10
    #10 0x7f608ce0782f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #11 0x556b1348ac18 in _start (/tmp/not-out/mp4+0xcac18)

DEDUP_TOKEN: rust_fuzzer_test_input--__rust_try--std::panicking::try::hfc14f47abd791d38
SUMMARY: MemorySanitizer: use-of-uninitialized-value /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20 in rust_fuzzer_test_input
  ORIGIN: invalid (0). Might be a bug in MemorySanitizer origin tracking.
    This could still be a bug in your code, too!
Unique heap origins: 35
Stack depot allocated bytes: 2392
Unique origin histories: 10
History depot allocated bytes: 240
Exiting
MS: 0 ; base unit: 0000000000000000000000000000000000000000


artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64: 
ERROR: 100% of fuzz targets seem to be broken. See the list above for a detailed information.
Check build failed.

[alexcrichton]

Ah sorry I don't think I'm knowledgeable enough about fuzzers/msan/etc to understand what's going on there. Given "Might be a bug in MemorySanitizer origin tracking." it looks like something possibly wasn't compiled correctly? I don't have much experience myself in using other fuzzers in Rust. It might be worthwhile to raise this on the cargo-fuzz repo and see if others can help out?