Skip to content

Conversation

dergoegge
Copy link

No description provided.

@dergoegge dergoegge changed the title [doc] Switch fuzzing docs to CMake doc: Switch fuzzing docs to CMake Jan 31, 2024
doc/fuzzing.md Outdated
$ cmake -S .. \
-DCMAKE_C_COMPILER="clang" \
-DCMAKE_CXX_COMPILER="clang++" \
-DCMAKE_BUILD_MODE=None \
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be in here or can i just drop it?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's wait for the discussion in #82.

Copy link

@pablomartin4btc pablomartin4btc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK 6f05459

Tested on Ubuntu 22.04.

Found an issue trying to build it following the libFuzzer section, perhaps it's my setup but on the bitcoin repo following the current guide using make works fine for me.
mkdir fuzz_build && cd fuzz_build
cmake -S .. \
   -DCMAKE_C_COMPILER="clang" \
   -DCMAKE_CXX_COMPILER="clang++" \
   -DCMAKE_BUILD_MODE=None \
   -DFUZZ=ON \
   -DSANITIZERS=undefined,address,fuzzer

cmake --build . -j$(nproc)

...
[ 10%] Linking CXX executable object
/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x23
CMakeFiles/unitester.dir/test/unitester.cpp.o: in function `main':
unitester.cpp:(.text.main[main]+0x0): multiple definition of `main'; /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a(FuzzerMain.cpp.o):(.text.main+0x0): first defined here
[ 11%] Building CXX object src/test/util/CMakeFiles/test_util.dir/blockfilter.cpp.o
[ 11%] Building CXX object src/test/util/CMakeFiles/test_util.dir/coins.cpp.o
[ 11%] Building CXX object src/test/util/CMakeFiles/test_util.dir/index.cpp.o
/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x23
CMakeFiles/object.dir/test/object.cpp.o: in function `main':
object.cpp:(.text.main[main]+0x0): multiple definition of `main'; /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a(FuzzerMain.cpp.o):(.text.main+0x0): first defined here
/usr/bin/ld: /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a(FuzzerMain.cpp.o): in function `main':
(.text.main+0x12): undefined reference to `LLVMFuzzerTestOneInput'
[ 11%] Building CXX object src/CMakeFiles/bitcoin_cli.dir/compat/stdin.cpp.o
[ 11%] Building CXX object src/CMakeFiles/bitcoin_cli.dir/rpc/client.cpp.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [src/univalue/CMakeFiles/unitester.dir/build.make:98: src/univalue/unitester] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1470: src/univalue/CMakeFiles/unitester.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 12%] Building CXX object src/test/util/CMakeFiles/test_util.dir/logging.cpp.o
[ 12%] Linking CXX static library libbitcoin_cli.a
[ 12%] Building CXX object src/test/util/CMakeFiles/test_util.dir/json.cpp.o
[ 12%] Built target bitcoin_cli
[ 12%] Building CXX object src/test/util/CMakeFiles/test_util.dir/mining.cpp.o
/usr/bin/ld: /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a(FuzzerMain.cpp.o): in function `main':
(.text.main+0x12): undefined reference to `LLVMFuzzerTestOneInput'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [src/univalue/CMakeFiles/object.dir/build.make:98: src/univalue/object] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1496: src/univalue/CMakeFiles/object.dir/all] Error 2
[ 12%] Building CXX object src/test/util/CMakeFiles/test_util.dir/net.cpp.o
[ 12%] Linking CXX static library libcrc32c.a
[ 12%] Built target crc32c
[ 12%] Built target generate_build_info
[ 12%] Building CXX object src/test/util/CMakeFiles/test_util.dir/random.cpp.o
[ 12%] Building CXX object src/test/util/CMakeFiles/test_util.dir/script.cpp.o
[ 13%] Building CXX object src/test/util/CMakeFiles/test_util.dir/setup_common.cpp.o
[ 13%] Building CXX object src/test/util/CMakeFiles/test_util.dir/str.cpp.o
[ 13%] Building CXX object src/test/util/CMakeFiles/test_util.dir/transaction_utils.cpp.o
[ 13%] Building CXX object src/test/util/CMakeFiles/test_util.dir/txmempool.cpp.o
[ 14%] Building CXX object src/test/util/CMakeFiles/test_util.dir/validation.cpp.o
[ 14%] Building CXX object src/test/util/CMakeFiles/test_util.dir/__/__/wallet/test/util.cpp.o
[ 14%] Built target minisketch_clmul
[ 14%] Linking C static library libsecp256k1.a
[ 14%] Built target secp256k1
[ 14%] Linking CXX static library libtest_util.a
[ 14%] Built target test_util
gmake: *** [Makefile:146: all] Error 2
...

I've also tried:

cmake -B fuzz_build \
   -DCMAKE_C_COMPILER="clang" \
   -DCMAKE_CXX_COMPILER="clang++" \
   -DCMAKE_BUILD_MODE=None \
   -DFUZZ=ON \
   -DSANITIZERS=undefined,address,fuzzer

cmake --build fuzz_build -j$(nproc)

[ 14%] Linking CXX executable unitester
[ 14%] Building CXX object CMakeFiles/minisketch.dir/src/minisketch/src/fields/generic_4bytes.cpp.o
/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x23
CMakeFiles/unitester.dir/test/unitester.cpp.o: in function `main':
unitester.cpp:(.text.main[main]+0x0): multiple definition of `main'; /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a(FuzzerMain.cpp.o):(.text.main+0x0): first defined here
[ 14%] Building CXX object CMakeFiles/minisketch.dir/src/minisketch/src/fields/generic_5bytes.cpp.o
[ 14%] Building CXX object CMakeFiles/leveldb.dir/src/leveldb/db/table_cache.cc.o
[ 14%] Building CXX object CMakeFiles/minisketch.dir/src/minisketch/src/fields/generic_6bytes.cpp.o
/usr/bin/ld: /usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a(FuzzerMain.cpp.o): in function `main':
(.text.main+0x12): undefined reference to `LLVMFuzzerTestOneInput'
[ 14%] Building CXX object CMakeFiles/leveldb.dir/src/leveldb/db/version_edit.cc.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [src/univalue/CMakeFiles/unitester.dir/build.make:98: src/univalue/unitester] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1470: src/univalue/CMakeFiles/unitester.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....

@hebasto
Copy link
Owner

hebasto commented Mar 8, 2024

@pablomartin4btc

Tested on Ubuntu 22.04.
Found an issue trying to build it following the libFuzzer section, perhaps it's my setup but on the bitcoin repo following the current guide using make works fine for me.

Does #115 fix this issue for you?

Copy link

@pablomartin4btc pablomartin4btc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pablomartin4btc

Tested on Ubuntu 22.04.
Found an issue trying to build it following the libFuzzer section, perhaps it's my setup but on the bitcoin repo following the current guide using make works fine for me.

Does #115 fix this issue for you?

Yes, Sir.

tACK #115, I can confirm the bug found above is fixed with it, please rebase with it.

Configure summary shows only fuzz binary is enabled, as expected.
Configure summary
=================
Executables:
  bitcoind ............................ OFF
  bitcoin-cli ......................... OFF
  bitcoin-tx .......................... OFF
  bitcoin-util ........................ OFF
  bitcoin-wallet ...................... OFF
Wallet support:
  SQLite, descriptor wallets .......... ON
  Berkeley DB, legacy wallets ......... OFF
Optional packages:
  GUI ................................. OFF
  external signer ..................... ON
  NAT-PMP ............................. OFF
  UPnP ................................ OFF
  ZeroMQ .............................. OFF
  USDT tracing ........................ ON
Tests:
  test_bitcoin ........................ OFF
  bench_bitcoin ....................... OFF
  fuzz binary ......................... ON

Shouldn't optional external signer be off as well?

Ran `fuzz binary successfully.
FUZZ=process_message src/test/fuzz/fuzz
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 465258960
INFO: Loaded 1 modules   (386634 inline 8-bit counters): 386634 [0x556259601c20, 0x55625966026a), 
INFO: Loaded 1 PC tables (386634 PCs): 386634 [0x556259660270,0x556259c46710), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED cov: 2713 ft: 2713 corp: 1/1b exec/s: 0 rss: 211Mb
#5	NEW    cov: 2714 ft: 2718 corp: 2/3b lim: 4 exec/s: 0 rss: 212Mb L: 2/2 MS: 3 CrossOver-CopyPart-InsertByte-
#11	NEW    cov: 2714 ft: 2719 corp: 3/5b lim: 4 exec/s: 0 rss: 212Mb L: 2/2 MS: 5 CopyPart-CopyPart-CrossOver-ChangeBit-InsertByte-
#14	NEW    cov: 2716 ft: 2770 corp: 4/6b lim: 4 exec/s: 0 rss: 212Mb L: 1/2 MS: 3 EraseBytes-ChangeBit-ChangeByte-
#17	NEW    cov: 2716 ft: 2771 corp: 5/8b lim: 4 exec/s: 0 rss: 212Mb L: 2/2 MS: 3 CrossOver-CrossOver-ChangeByte-
#28	NEW    cov: 2717 ft: 2773 corp: 6/9b lim: 4 exec/s: 0 rss: 212Mb L: 1/2 MS: 1 ChangeBit-
#31	REDUCE cov: 2717 ft: 2773 corp: 6/8b lim: 4 exec/s: 0 rss: 212Mb L: 1/2 MS: 3 EraseBytes-ChangeByte-ChangeByte-
#36	NEW    cov: 2717 ft: 2774 corp: 7/12b lim: 4 exec/s: 0 rss: 212Mb L: 4/4 MS: 5 EraseBytes-ChangeBit-InsertByte-InsertByte-InsertByte-
#82	REDUCE cov: 2718 ft: 2779 corp: 8/15b lim: 4 exec/s: 0 rss: 213Mb L: 3/4 MS: 1 CMP- DE: "\000\000"-
#111	REDUCE cov: 2718 ft: 2779 corp: 8/14b lim: 4 exec/s: 0 rss: 214Mb L: 2/4 MS: 4 ChangeASCIIInt-ChangeByte-ChangeBinInt-EraseBytes-
#127	NEW    cov: 2718 ft: 2781 corp: 9/18b lim: 4 exec/s: 0 rss: 214Mb L: 4/4 MS: 1 PersAutoDict- DE: "\000\000"-
#193	NEW    cov: 2718 ft: 2783 corp: 10/19b lim: 4 exec/s: 0 rss: 215Mb L: 1/4 MS: 1 CrossOver-
#282	REDUCE cov: 2718 ft: 2783 corp: 10/18b lim: 4 exec/s: 0 rss: 216Mb L: 3/4 MS: 4 ChangeBit-ChangeASCIIInt-ChangeByte-EraseBytes-
#609	REDUCE cov: 2718 ft: 2783 corp: 10/17b lim: 6 exec/s: 0 rss: 220Mb L: 2/4 MS: 2 EraseBytes-ChangeBit-
#780	REDUCE cov: 2718 ft: 2783 corp: 10/16b lim: 6 exec/s: 0 rss: 222Mb L: 1/4 MS: 1 EraseBytes-
#820	NEW    cov: 2718 ft: 2784 corp: 11/22b lim: 6 exec/s: 0 rss: 223Mb L: 6/6 MS: 5 CrossOver-ChangeByte-EraseBytes-ChangeBit-InsertRepeatedBytes-
#926	NEW    cov: 2719 ft: 2785 corp: 12/26b lim: 6 exec/s: 926 rss: 224Mb L: 4/6 MS: 1 InsertRepeatedBytes-
#997	REDUCE cov: 2719 ft: 2785 corp: 12/24b lim: 6 exec/s: 997 rss: 226Mb L: 2/6 MS: 1 EraseBytes-
#1166	NEW    cov: 2719 ft: 2787 corp: 13/30b lim: 6 exec/s: 1166 rss: 228Mb L: 6/6 MS: 4 InsertRepeatedBytes-CrossOver-ChangeBinInt-CMP- DE: "addrv2"-
#1182	REDUCE cov: 2719 ft: 2787 corp: 13/28b lim: 6 exec/s: 1182 rss: 228Mb L: 4/6 MS: 1 CrossOver-
#1414	REDUCE cov: 2719 ft: 2788 corp: 14/36b lim: 8 exec/s: 1414 rss: 231Mb L: 8/8 MS: 2 ShuffleBytes-CopyPart-
#1427	REDUCE cov: 2719 ft: 2788 corp: 14/35b lim: 8 exec/s: 1427 rss: 231Mb L: 5/8 MS: 3 CopyPart-EraseBytes-InsertByte-
#1689	REDUCE cov: 2719 ft: 2788 corp: 14/34b lim: 8 exec/s: 1689 rss: 235Mb L: 7/7 MS: 2 ShuffleBytes-EraseBytes-
#1800	REDUCE cov: 2719 ft: 2788 corp: 14/33b lim: 8 exec/s: 1800 rss: 237Mb L: 1/7 MS: 1 EraseBytes-
...

hebasto added a commit that referenced this pull request Mar 9, 2024
…nabled

07b01dd fixup! cmake: Build `univalue` static library (Hennadii Stepanov)

Pull request description:

  This PR addresses #89 (review).

ACKs for top commit:
  pablomartin4btc:
    tACK 07b01dd

Tree-SHA512: 7bf0abd41d606b71b65fcf0e6aa32387201c6ec67269abefaf37b160edbe1ce91f3de7e61d166b136205a4dc5c828599c3eb62273d1286ddd570472a4d05f6ff
@hebasto
Copy link
Owner

hebasto commented Mar 9, 2024

Shouldn't optional external signer be off as well?

Is it off on the master branch?

@pablomartin4btc
Copy link

pablomartin4btc commented Mar 9, 2024

Shouldn't optional external signer be off as well?

Is it off on the master branch?

Yes.
That was on the fix (#115) branch (240308-cmake-CG).

On master following autotooling instructions from the current fuzzing doc:

./autogen.sh
CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined
Options used to compile and link:
  external signer = no
  multiprocess    = no
  with libs       = no
  with wallet     = yes
    with sqlite   = yes
    with bdb      = no
  with gui / qt   = no
  with zmq        = no
  with test       = not building test_bitcoin because fuzzing is enabled
  with fuzz binary = yes
  with bench      = no
  with upnp       = no
  with natpmp     = no
  USDT tracing    = yes
  sanitizers      = address,fuzzer,undefined
  debug enabled   = no
  gprof enabled   = no
  werror          = no

USDT tracing option is ON on both, master and cmake.

But I mentioned because during configuration user can see this legend:

CMake Warning at CMakeLists.txt:112 (message):
  FUZZ=ON will disable all other targets and force BUILD_FUZZ_BINARY=ON.

I got confused with targets/ options, perhaps we can clarify "default/ edit: 'fuzzing required (?)' options x, y z, will remain on", or something like that, but maybe it's already clear and I missed it.

@pablomartin4btc
Copy link

This needs to be rebased.

@dergoegge
Copy link
Author

Rebased

hebasto added a commit that referenced this pull request Mar 25, 2024
d96f51a fixup! cmake: Add fuzzing options (Hennadii Stepanov)

Pull request description:

  The behavior is aligned with the master branch.

  First noticed in #89 (review).

ACKs for top commit:
  dergoegge:
    utACK d96f51a

Tree-SHA512: 9ba8ab9c4c14814f09c583e28c8bd3e0c8bc2d2ab83b79316b771ef376cb1dd4b8de8bbbd0777367d5e8510332471353da95242d74434d6b30bfdb8e285b2efc
@hebasto hebasto added the documentation Docs and manuals label Apr 27, 2024
@hebasto hebasto force-pushed the cmake-staging branch 2 times, most recently from f918ccb to 22270ee Compare May 4, 2024 13:15
@dergoegge dergoegge force-pushed the 2024-01-cmake-fuzzing-docs branch 4 times, most recently from 8dc9b90 to aa531e9 Compare May 21, 2024 13:00
@dergoegge dergoegge force-pushed the 2024-01-cmake-fuzzing-docs branch from aa531e9 to 1023722 Compare May 21, 2024 13:14
Copy link
Owner

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 1023722, changes look sane, did not test them.

@hebasto hebasto merged commit 63f15c4 into hebasto:cmake-staging May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Docs and manuals

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants