Improve the speed of the benchmarks

[jvoisin]

Currently the CI takes around 90min to run: this is a bit excessive. Here is what we could do to reduce a bit this number:

• Change the parameters for the redis benchmark: SYSMALLOC=1 /__w/mimalloc-bench/mimalloc-bench/extern/redis-6.2.6/src/redis-benchmark -r 1000000 -n 1000000 -q -P 16 lpush a 1 2 3 4 5 lrange a 1 5 takes around a full minute, given that we have ~16 allocators, it adds up rapidly.
  • Done in 410f665
• Lean takes around 1-3 minutes per allocator, and is unmaintained. Shall we try to move to lean4 ?

[mjp41]

So my wondering was about how to split the work up into more parallel tasks.

We could build a Docker image with all the benchmarks, and then we can install that image, and run build and test a subset of the allocators. This would allow us to increase the parallelism, without too much duplication of cost. It would provide a more stable test setup for replication.

I am imagining the allocators change more frequently than the benchmarks for this to make sense. But based on the PRs I think that is correct.

We would still want to test that we can build all the benchmarks, but that could be a separate task, and not have the allocator testing depend on it.

[jvoisin]

Unfortunately, I don't think that github actions has the concept of artifacts like gitlab does :/

[derSteFfi]

I am kind of in line with @mjp41 here. Currently, our CI is running every benchmark against every allocator, once. This basically means we do a full benchmark run in CI. imho, that is not what CI is for (at least not in this project). For us, it should be sufficient to:

1. build all benchmarks
2. build all allocators
3. run all benchmarks with the system allocator to ensure they built properly
4. run all allocators against a short benchmark to ensure they built properly

The short benchmark might be cfrac, barnes (is suspiciously constant in the time needed - do they even use malloc?), gs or another one that does use relevant parts of the allocators but does not take too long.
Our current Ubuntu setup takes ~11 minutes to build everything, cfrac runs in under 10 seconds for most allocators, a complete run of all tests (allt) with the system allocator takes about a minute on my machine, so I estimate around 2 minutes on the CI. Given 21 allocators on Ubuntu right now, about 4 minutes of cfrac running has to be added. This brings us to a total of less than 20 minutes with the setup being the biggest block - we can improve that further down the road.

Considering lean4: I'm currently building the release version on a machine quite a lot stronger than CI - I'm 20 minutes in and still not done. So I think that would not be a good idea.

Another detail: mi-dbg and mi2-dbg take quite some longer time (understandable). Do we need them in the CI? It seems they're suited if you would develop mimalloc, but not really suited for benchmarking... I'm not advocating for removing them, but not including them in the CI. They also spam the console pretty badly.

TLDR: I'd suggest the following:

1. remove mi-dbg and mi2-dbg from the CI
2. reduce the test runs: once with sys allt, once with alla cfrac
3. include all tests that can be built successfully in allt
4. as a separate but connected issue: improve the speed of the setup. maybe through Proposal: port build-bench-env.sh to make #230 , maybe by other means.

[mjp41]

So I have been experimenting with some level of caching today. But the structure of build-bench-env.sh has everything all in one. So for instance have a Dockerfile like:

FROM ubuntu:24.04

# Pull mimalloc-bench
RUN  mkdir build
RUN mkdir -p /build/mimalloc-bench
COPY . /build/mimalloc-bench

# Install dependencies
WORKDIR /build/mimalloc-bench
RUN ./build-bench-env.sh packages

# Build benchmarks
RUN ./build-bench-env.sh bench

RUN ./build-bench-env.sh redis

RUN ./build-bench-env.sh rocksdb

RUN ./build-bench-env.sh lean

# Bulid allocator
ARG allocator=mi
RUN ./build-bench-env.sh $allocator

# Run benchmarks
ARG benchs=allt
WORKDIR /build/mimalloc-bench/out/bench
RUN ../../bench.sh $allocator $benchs

Won't nicely cache as COPY will change if we change a single allocator version or a benchmark version. I am currently thinking for snmalloc's CI to build a docker container for the benchmarks, and inject snmalloc built outside the container, or built without build-bench-env, to be the thing that is tested.

[mjp41]

So I've experimented a bit more with Docker to build CI for snmalloc that uses mimalloc-bench.

microsoft/snmalloc#795

This uses Dockers caching to mostly pull the benchmarks from cache. It builds snmalloc from a different source, so changes to snmalloc don't affect the benchmark builds.

I wonder if we can restructure the build in mimalloc-bench to be able to reuse the same kind of caching to improve CI times.

I also used a cut down version of the Python scripts for graphs to generate a json file to submit to bencher.dev. This script is currently in the snmalloc PR but I would be keen to add to mimalloc-bench instead, so others could use it. This would also make copyright/licensing simpler. @jvoisin do you think it makes sense to add here?

[jvoisin]

I also used a cut down version of the Python scripts for graphs to generate a json file to submit to bencher.dev. This script is currently in the snmalloc PR but I would be keen to add to mimalloc-bench instead, so others could use it. This would also make copyright/licensing simpler. @jvoisin do you think it makes sense to add here?

Absolutely <3