More parallel tests

[Sjors]

When running make -j… check I noticed it doesn't use more than 3 CPU's at any one time, one for tests, exhaustive_tests and noverify_tests. At least when I tried it on an Intel macOS 13.2 machine.

This is a bottleneck when running the full Bitcoin Core test suite on something slow like with msan (tried on a Ubuntu 22.10 x86_64 machine using Docker).

That said, it's not the end of the world either, since I could also just run more test suite (Docker containers) in parallel.

[real-or-random]

When running make -j… check I noticed it doesn't use more than 3 CPU's at any one time, one for tests, exhaustive_tests and noverify_tests. At least when I tried it on an Intel macOS 13.2 machine.

Yep, the only parallelism we have is that one from different binaries.

As you say, it's not the end of the world, but it would be good to improve this if it's a bottleneck in Core. I thought a few times about this issue, but my pain was never big enough that I actually spend time working on this.

• Pragmatically speaking, we could just run 4x tests 16 and 4x noverify_tests 16 instead of 1x tests 64 and 1x noverify_tests 64. This means we'll run the deterministic part of the test suite 4x... Not elegant, but also not the end of the world. It errs on the other side of the spectrum.
• Some "proper" unit testing framework would give us a way to run different test cases in different processes/threads. tests: allow user to select tests via command line args #1211 is a bit of a poor man's approach version of this, and it is ready right now. This will make it possible to run different subsets in different processes. One disadvantage is that we'll need to run to split the test suite manually and find the right command line args. It would perhaps make sense to add more generic arguments like "--split 4 --part 3", which would split the test cases into four groups and run only the third of it.

[apoelstra]

This is also a pain-point for me -- with a 32-core machine, the bottlenecks in building Core are ./configure and the secp tests.

I like the idea of running ./tests 16 four times (or maybe we could be a little bit smart and divide 64 by the -j value). Yes, it's an inelegant hack, but it can also be done extremely quickly and backed out when we have a better solution.

[Sjors]

Supporting -j is probably the easiest way to hook into Bitcoin Core's existing CI system. This value is passed to make but also to the functional tests. It's probably also the most intuitive: make -j33 check

[apoelstra]

@Sjors FYI if you don't actually care to run the tests in full, you can set the environment SECP256K1_TEST_ITERS=1 (I think you can even set it to 0 ... we had a bug in the past from somebody doing that, so I guess we fixed it ... though even 0 will still run some tests).

[real-or-random]

(or maybe we could be a little bit smart and divide 64 by the -j value

I checked, and I don't think there's no way to access the -j value from within the Makefile.

But yeah, we could still just use 4x16.

[hebasto]

Supporting -j is probably the easiest way to hook into Bitcoin Core's existing CI system. This value is passed to make but also to the functional tests. It's probably also the most intuitive: make -j33 check

For example: #1211 (review).