Builds that takes more than 4 hours

[petrasvestartas]

Hi,

I have a build for OpenCascade Kernel that takes:

• Windows 20 minutes to build.
• Mac 40 min
• Ubuntu 20 min

This is for one python version only.
If I build for Python 3.9, 3.10, 3.11 and 3.12 ABI this will highly likely exceed what Github CI time limit.

Could you help me to split these github actions into jobs for each python version?

https://github.com/petrasvestartas/compas_occt/blob/main/.github/workflows/release.yml
https://github.com/petrasvestartas/compas_occt/blob/main/pyproject.toml

This actions build python 3.9 to 3.11 and does not build 3.12:
https://github.com/petrasvestartas/compas_occt

name: Release

on:
  push:
    tags:
      - "v*"  # Runs only when a version tag (e.g., v1.0.0) is pushed.

jobs:
  create_release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Create GitHub Release
        id: create_release
        uses: actions/create-release@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          tag_name: ${{ github.ref }}
          release_name: Release ${{ github.ref }}
          draft: false
          prerelease: false

  build_wheels:
    name: Build ${{ matrix.python }} wheels on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        python: ["cp39", "cp310", "cp311", "cp312"]
        os: [ubuntu-latest, macos-latest, windows-latest]
        include:
          - os: ubuntu-latest
            platform: manylinux
          - os: macos-latest
            platform: mac
          - os: windows-latest
            platform: windows

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install cibuildwheel
        run: pipx install cibuildwheel==2.23.1

      - name: Build wheels
        run: cibuildwheel --output-dir wheelhouse .
        env:
          CIBW_BUILD: ${{ matrix.python }}-*
          CIBW_ARCHS: auto64
          # Set OCCT build to use fewer cores to avoid resource exhaustion
          CMAKE_BUILD_PARALLEL_LEVEL: 2

      - uses: actions/upload-artifact@v4
        with:
          name: wheels-${{ matrix.platform }}-${{ matrix.python }}
          path: wheelhouse/*.whl

  build_sdist:
    name: Build source distribution
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Build SDist
        run: pipx run build --sdist

      - uses: actions/upload-artifact@v4
        with:
          name: sdist
          path: dist/*.tar.gz

  publish:
    needs: [build_sdist, build_wheels]
    runs-on: ubuntu-latest
    environment: pypi
    # The URL is created after successful deploy
    # url: https://pypi.org/project/compas_occt
    permissions:
      id-token: write  # Required for PyPI trusted publishing

    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: wheels-*
          path: dist
          merge-multiple: true

      - uses: actions/download-artifact@v4
        with:
          name: sdist
          path: dist

      - name: List files before upload
        run: ls -lhR dist

      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        # Comment out the following line to publish to PyPI instead of TestPyPI
        with:
          repository-url: https://test.pypi.org/legacy/

Thank you

[LecrisUT]

I am afraid that is the best you can do w.r.t. github actions splitting.

You could make a design where the non-python parts can be built as an independent project and pre-build that and share it across the python builds. The wheel repairs should take care of pulling in and linking the dependencies.

[petrasvestartas]

Your previous comment to split the the mac versions into mac 13 and 14 helped greatly, this is the solution for github release:

name: Release

on:
  push:
    tags:
      - "v*"  # Runs only when a version tag (e.g., v1.0.0) is pushed.

jobs:
  create_release:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          
      - name: Create GitHub Release
        id: create_release
        uses: softprops/action-gh-release@v1
        with:
          tag_name: ${{ github.ref_name }}
          name: Release ${{ github.ref_name }}
          draft: false
          prerelease: false
          generate_release_notes: true

  build_wheels:
    name: Build wheels on ${{ matrix.platform }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        include:
          - os: ubuntu-latest
            platform: manylinux
          # - os: macos-latest
          #   platform: mac
          - os: macos-13
            platform: mac-intel
          - os: macos-14
            platform: mac-arm
          - os: windows-latest
            platform: windows

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install cibuildwheel
        run: pipx install cibuildwheel==2.23.1

      - name: Build wheels
        run: cibuildwheel --output-dir wheelhouse .

      - uses: actions/upload-artifact@v4
        with:
          name: wheels-${{ matrix.platform }}
          path: wheelhouse/*.whl

  build_sdist:
    name: Build source distribution
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Build SDist
        run: pipx run build --sdist

      - uses: actions/upload-artifact@v4
        with:
          name: sdist
          path: dist/*.tar.gz

  publish:
    needs: [build_sdist, build_wheels]
    runs-on: ubuntu-latest
    environment: pypi
    # The URL is created after successful deploy
    # url: https://pypi.org/project/compas_occt
    permissions:
      id-token: write  # Required for PyPI trusted publishing

    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: wheels-*
          path: dist
          merge-multiple: true

      - uses: actions/download-artifact@v4
        with:
          name: sdist
          path: dist

      - name: List files before upload
        run: ls -lhR dist

      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        # Comment out the following line to publish to PyPI instead of TestPyPI
        with:
          repository-url: https://test.pypi.org/legacy/