Skip to content

make CI jobs faster, and other improvements #181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 58 commits into from

Conversation

vszakats
Copy link
Member

@vszakats vszakats commented Jul 29, 2025

After this patch BuildFuzzers and Testi386 jobs need about 7.5 minutes
per job, down from 15-16m, for a >2x speed-up. There should noticeable
improvements in the other jobs, too:

  • Mainline (address): 4m45s → 3m30s (gain: 1m15s)
  • Mainline (memory): 2m45s → 2m0s (gain: 45s)
  • JustDependencies: 2m → 1m50s (gain: 10s)

Before: https://github.com/curl/curl/actions/runs/16769657727
After: https://github.com/curl/curl-fuzzer/actions/runs/16775268062?pr=181

Also reduce log noise and fix minor issues found along the way. Add
Ninja support.

Details:

  • drop progress bar in apt commands.
  • CI: move env: before run: to match control flow. (style)
  • CI and scripts/ossfuzzdeps.sh: install ninja. For more robust build
    performance across platforms, and more readable logs.
  • cmake: drop BUILD_COMMAND $(MAKE) and
    INSTALL_COMMAND $(MAKE) install and use the cmake defaults.
    The defaults work with both GNU Make and Ninja generators. This also
    avoids passing a GNU Make macro to the generator, which broke with and
    error when used with Ninja.
    Ref: https://cmake.org/cmake/help/latest/module/ExternalProject.html
  • replace one remaining $(MAKE) use with ${MAKE}, and initialize
    the latter from the calling env. This is necessary to keep avoiding
    the super lengthy OpenSSL manual install phase, which needs a custom
    install target (install_sw) unfortunately.
  • cmake: hide download progress lines to reduce log noise.
  • cmake: specify build byproducts in ExternalProject_Add() commands.
    Necessary for Ninja to figure out what output is generated, to resolve
    them as dependencies.
  • replace gzip packages with .tar.xz where available, for smaller
    downloads.
  • switch nghttp2 build from autotools to cmake for better performance.
    (esp. better configure performance.)
  • drop redundant zlib dependency for nghttp2. (lib-only builds don't use
    it).
  • disable unnecessary components in dependencies to save build time. In
    openssl, zstd, nghttp2, libidn2. (e.g. tools, docs, examples, tests.)
  • drop redundant option --enable-websockets when building curl.
    It became the default after leaving the experimental status.
  • disable dependency tracking in autotools builds for better
    performance.
  • fix MAKEFLAGS, which became -j4-j4 during ossfuzz builds.
    This made all builds run without parallelism.
    Also show this env in the log, for visibility.
  • switch curl build from autotools to cmake for better performance.
  • stop building the unused curl tool to finish builds faster.
  • curl: drop redundant DOWNLOAD_EXTRACT_TIMESTAMP option for Git.
  • cmake: drop redundant BUILD_IN_SOURCE and SOURCE_SUBDIR
    options.
  • restore groff-base. Turns out to be a tiny patch; it's not worth
    the manual workaround.
    Follow-up to d0bf19b
  • remove exec attribute from a C++ source.
  • ossfuzz: drop redundant MAKEFLAGS settings and env dumps.
  • CI: skip updating man-db for faster package installs. (Ubuntu)
  • use -s in MAKEFLAGS to keep the low verbosity of sub-builds as
    before this patch. Without it, CMake seems to add -w when using
    Ninja.

Due to the way ossfuzz works (?), these changes cannot be tested within
this PR, because part of the repo is pulled from curl/curl-fuzzer master
while testing changes pushed in this PR. This makes the scripts miss to
install Ninja. This setup surprised me. Also not fully from curl
upstream, because some of the parts seem to always come from master.

Helper patch: d0bf19b
Partial test from curl: curl/curl#18202
Bug: curl/curl#18140 (comment)


  • switch curl build from autotools to cmake. For better performance.
  • stop building the curl tool to save time.

Possible future improvements?:

@vszakats vszakats marked this pull request as draft July 29, 2025 01:09
@vszakats vszakats changed the title compile_target.sh: try bumping make -j, switching to ninja compile_target.sh: try to improve performance Aug 5, 2025
@vszakats vszakats changed the title compile_target.sh: try to improve performance try to improve performance Aug 5, 2025
@vszakats vszakats marked this pull request as ready for review August 6, 2025 01:34
vszakats added a commit that referenced this pull request Aug 6, 2025
To test #181. The ossfuzz CI job seems to pull `scripts/ossfuzzdeps.sh`
from the master branch while running on a PR. This patch is an attempt
to modify that file to finish testing said PR.

In case of issues, I revert or follow up.

Ref: #181
vszakats added 23 commits August 6, 2025 04:53
CMake Error:
  Running

   '/usr/local/bin/ninja' '-C' '/home/runner/work/curl-fuzzer/curl-fuzzer/build' '-t' 'recompact'

  failed with:

   ninja: error: build.ninja:1174: bad $-escape (literal $ must be written as $$)

CMake Generate step failed.  Build files cannot be regenerated correctly.

https://github.com/curl/curl-fuzzer/actions/runs/16584395005/job/46906820771?pr=181
```
CMake Error:
  Running

   '/usr/local/bin/ninja' '-C' '/home/runner/work/curl-fuzzer/curl-fuzzer/build' '-t' 'recompact'

  failed with:

   ninja: error: build.ninja:1174: bad $-escape (literal $ must be written as $$)
```
https://github.com/curl/curl-fuzzer/actions/runs/16755832030/job/47438071308#step:4:57
… the orchestrator which is incompatible with ninja due to using Externalproject
wget already installed, suggestions/recommendations not install
in the docker machine by default.
@vszakats vszakats changed the title try to improve performance make BuildFuzzers job twice as fast, and other improvements Aug 6, 2025
This reverts commit d321b8b.

BuildFuzzers went up from 7.5 minutes to 10.
@vszakats
Copy link
Member Author

vszakats commented Aug 6, 2025

@cmeister2 What do you think of these updates?

Before this patch the GNU make command-line was:
/usr/bin/gmake -s -j4 --jobserver-auth=3,4

After moving to Ninja this changed implicitly to:
make -w -j4 --jobserver-auth=3,4

as seen in JustDependencies and Mainline jobs.
no advantage in micro-managing MAKE
@vszakats vszakats changed the title make BuildFuzzers job twice as fast, and other improvements make BuildFuzzers jobs twice as fast, and other improvements Aug 6, 2025
Copy link
Collaborator

@cmeister2 cmeister2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broadly happy but I would like the commits to be squashed or otherwise rebased into logical blocks when merging.

@vszakats
Copy link
Member Author

vszakats commented Aug 6, 2025

I always squash before merge, as the last step, to not lose track of the local history till there may be a need for changes.

The total diff in the meantime: https://github.com/curl/curl-fuzzer/pull/181.diff

(In this case I'm not keen to split into multiple PRs, because the number of touched lines is fairly small, but also very prone to conflicts if torn apart.)

@vszakats vszakats changed the title make BuildFuzzers jobs twice as fast, and other improvements make CI jobs faster, and other improvements Aug 6, 2025
@vszakats vszakats closed this in 2000455 Aug 6, 2025
@vszakats vszakats deleted the tryopt branch August 6, 2025 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants