`make test` oversubscribes threads

[oscardssmith]

Due to some combination of Julia threads not knowing about BLAS threads (and possibly something else), I'm seeing make test bring my cpu (r5-3600) to a load average of 16. This is likely dramatically slowing down our CI. Could we manually set BLAS threads to 1 for our tests? I think it would resolve the issue, and lead to tests completing faster.

[inkydragon]

prev_num_threads = BLAS.get_num_threads()
BLAS.set_num_threads(1)

# do something

BLAS.set_num_threads(prev_num_threads)

Copy from #41785

[DilumAluthge]

For people running the Julia test suite locally, I think it's fine to have an option to set the number of BLAS threads used during the test suite. Perhaps it could be an optional keyword argument to the Base.runtests function. The default value could be nothing, in which case the current behavior is preserved.

However, for CI, I would think that we do want to make sure that we are testing the BLAS multithreaded code paths, so I don't know if it makes sense to disable BLAS multithreading on CI. @staticfloat any thoughts?

[chriselrod]

You could set the environmental variable OPENBLAS_NUM_THREADS=1 before running make test.

[staticfloat]

I think we want all of the following things:

1. Code coverage; every make test should test both single and multi-threaded codepaths in Julia and OpenBLAS.

2. Proper core loading; In general, we want to launch just enough threads to saturate our cores, and no more. In particular, this means that BLAS operations on workers when doing a full make test should not be using more than 1 core.

3. Fast tests when running a test in isolation; if I want to test just the LinearAlgebra/lu suite, we might want to support allowing BLAS to run multithreaded there.

And so, I think what we need is the following:

1. A test suite that ignores global settings and forces OpenBLAS to use lots of cores for a few quick tests to ensure that this still works; not exhaustive tests, just a quick smoke test.

2. When launching tests, we set OPENBLAS_NUM_THREADS=1 inside of Base.runtests() if we are launching multiple simultaneous workers.

3. Exhaustive LinearAlgebra tests should be done in a separate LinearAlgebra stdlib repository; their testing cadence should have basically nothing to do with our tests, and we are wasting a huge amount of CI time re-testing basic BLAS operations when the only thing we've changed is something in Sockets. :P