Refactor build pipeline to reduce duplicate builds

[leighmcculloch]

What

Refactor build pipeline to:

1. Make a single location for updating software versions.
2. Make all the dependency software build first for all images, deduplicating them so that it builds each dependency once, then builds the quickstart images.

The new workflow also comes with some bonus features:

• It supports scheduled builds, such as a nightly-like image build.
• It can be triggered via a workflow call from another repository. Another repository can use the quickstart repo to build any quickstart with any configuration of software, and to run quickstarts standard "can everything start up and sync" tests. That resulting image can be used in a subsequent test.
• It can accept branch references for software components and handle them stably across the build pipeline.

Old Workflow

flowchart TD
    subgraph Main Branch Workflow
        subgraph Latest Pipeline
            subgraph Latest AMD64 Deps Pipeline
                B1_latest_amd64[xdr] --> B_latest_amd64
                B2_latest_amd64[core] --> B_latest_amd64
                B3_latest_amd64[rpc] --> B_latest_amd64
                B4_latest_amd64[horizon] --> B_latest_amd64
                B5_latest_amd64[friendbot] --> B_latest_amd64
                B6_latest_amd64[lab] --> B_latest_amd64
            end

            B_latest_amd64[load/prepare/build deps amd64] --> C_latest_amd64[build latest quickstart amd64]
            C_latest_amd64 --> D_latest_amd64[test latest amd64]
            D_latest_amd64 --> E_latest_amd64[push latest amd64]
            E_latest_amd64 --> F_latest[push-manifest latest amd64+arm64]
            F_latest --> G_latest[action latest amd64+arm64]

            subgraph Latest ARM64 Deps Pipeline
                B1_latest_arm64[xdr] --> B_latest_arm64
                B2_latest_arm64[core] --> B_latest_arm64
                B3_latest_arm64[rpc] --> B_latest_arm64
                B4_latest_arm64[horizon] --> B_latest_arm64
                B5_latest_arm64[friendbot] --> B_latest_arm64
                B6_latest_arm64[lab] --> B_latest_arm64
            end

            B_latest_arm64[load/prepare/build deps arm64] --> C_latest_arm64[build latest quickstart arm64]
            C_latest_arm64 --> D_latest_arm64[test latest arm64]
            D_latest_arm64 --> E_latest_arm64[push latest arm64]
            E_latest_arm64 --> F_latest
        end

        subgraph Testing Pipeline
            subgraph Testing AMD64 Deps Pipeline
                B1_testing_amd64[xdr] --> B_testing_amd64
                B2_testing_amd64[core] --> B_testing_amd64
                B3_testing_amd64[rpc] --> B_testing_amd64
                B4_testing_amd64[horizon] --> B_testing_amd64
                B5_testing_amd64[friendbot] --> B_testing_amd64
                B6_testing_amd64[lab] --> B_testing_amd64
            end

            B_testing_amd64[load/prepare/build deps amd64] --> C_testing_amd64[build testing quickstart amd64]
            C_testing_amd64 --> D_testing_amd64[test testing amd64]
            D_testing_amd64 --> E_testing_amd64[push testing amd64]
            E_testing_amd64 --> F_testing[push-manifest testing amd64+arm64]
            F_testing --> G_testing[action testing amd64+arm64]

            subgraph Testing ARM64 Deps Pipeline
                B1_testing_arm64[xdr] --> B_testing_arm64
                B2_testing_arm64[core] --> B_testing_arm64
                B3_testing_arm64[rpc] --> B_testing_arm64
                B4_testing_arm64[horizon] --> B_testing_arm64
                B5_testing_arm64[friendbot] --> B_testing_arm64
                B6_testing_arm64[lab] --> B_testing_arm64
            end

            B_testing_arm64[load/prepare/build deps arm64] --> C_testing_arm64[build testing quickstart arm64]
            C_testing_arm64 --> D_testing_arm64[test testing arm64]
            D_testing_arm64 --> E_testing_arm64[push testing arm64]
            E_testing_arm64 --> F_testing
        end

        subgraph Future Pipeline
            subgraph Future AMD64 Deps Pipeline
                B1_future_amd64[xdr] --> B_future_amd64
                B2_future_amd64[core] --> B_future_amd64
                B3_future_amd64[rpc] --> B_future_amd64
                B4_future_amd64[horizon] --> B_future_amd64
                B5_future_amd64[friendbot] --> B_future_amd64
                B6_future_amd64[lab] --> B_future_amd64
            end

            B_future_amd64[load/prepare/build deps amd64] --> C_future_amd64[build future quickstart amd64]
            C_future_amd64 --> D_future_amd64[test future amd64]
            D_future_amd64 --> E_future_amd64[push future amd64]
            E_future_amd64 --> F_future[push-manifest future amd64+arm64]
            F_future --> G_future[action future amd64+arm64]

            subgraph Future ARM64 Deps Pipeline
                B1_future_arm64[xdr] --> B_future_arm64
                B2_future_arm64[core] --> B_future_arm64
                B3_future_arm64[rpc] --> B_future_arm64
                B4_future_arm64[horizon] --> B_future_arm64
                B5_future_arm64[friendbot] --> B_future_arm64
                B6_future_arm64[lab] --> B_future_arm64
            end

            B_future_arm64[load/prepare/build deps arm64] --> C_future_arm64[build future quickstart arm64]
            C_future_arm64 --> D_future_arm64[test future arm64]
            D_future_arm64 --> E_future_arm64[push future arm64]
            E_future_arm64 --> F_future
        end

        G_latest --> H[complete]
        G_testing --> H
        G_future --> H
    end

Loading

New Workflow

flowchart TD
    subgraph New Workflow
        I[setup]

        subgraph Deps Pipeline
            I --> J1[xdr amd64]
            I --> J2[xdr arm64]
            I --> J3[core amd64]
            I --> J4[core arm64]
            I --> J5[rpc amd64]
            I --> J6[rpc arm64]
            I --> J7[horizon amd64]
            I --> J8[horizon arm64]
            I --> J9[friendbot amd64]
            I --> J10[friendbot arm64]
            I --> J11[lab amd64]
            I --> J12[lab arm64]
            J1 --> R
            J2 --> R
            J3 --> R
            J4 --> R
            J5 --> R
            J6 --> R
            J7 --> R
            J8 --> R
            J9 --> R
            J10 --> R
            J11 --> R
            J12 --> R
        end

        R[load/prepare/build deps]

        subgraph Latest Pipeline
            R --> L_latest_amd64[build latest quickstart amd64]
            R --> L_latest_arm64[build latest quickstart arm64]
            L_latest_amd64 --> M_latest_amd64[test latest amd64]
            L_latest_arm64 --> M_latest_arm64[test latest arm64]
            M_latest_amd64 --> N_latest_amd64[push latest amd64]
            M_latest_arm64 --> N_latest_arm64[push latest arm64]
            N_latest_amd64 --> O_latest[push-manifest latest amd64+arm64]
            N_latest_arm64 --> O_latest
            O_latest --> P_latest[action latest amd64+arm64]
        end

        subgraph Testing Pipeline
            R --> L_testing_amd64[build testing quickstart amd64]
            R --> L_testing_arm64[build testing quickstart arm64]
            L_testing_amd64 --> M_testing_amd64[test testing amd64]
            L_testing_arm64 --> M_testing_arm64[test testing arm64]
            M_testing_amd64 --> N_testing_amd64[push testing amd64]
            M_testing_arm64 --> N_testing_arm64[push testing arm64]
            N_testing_amd64 --> O_testing[push-manifest testing amd64+arm64]
            N_testing_arm64 --> O_testing
            O_testing --> P_testing[action testing amd64+arm64]
        end

        subgraph Future Pipeline
            R --> L_future_amd64[build future quickstart amd64]
            R --> L_future_arm64[build future quickstart arm64]
            L_future_amd64 --> M_future_amd64[test future amd64]
            L_future_arm64 --> M_future_arm64[test future arm64]
            M_future_amd64 --> N_future_amd64[push future amd64]
            M_future_arm64 --> N_future_arm64[push future arm64]
            N_future_amd64 --> O_future[push-manifest future amd64+arm64]
            N_future_arm64 --> O_future
            O_future --> P_future[action future amd64+arm64]
        end

        P_latest --> Q[complete]
        P_testing --> Q
        P_future --> Q
    end

Loading

Usage Building Custom Quickstarts

The new workflow has its build workflow separated from the push workflow so that other repos can call into the build workflow to build a custom quickstart for use in their own tests.

To run a quickstart that exists in DockerHub, do as you can do today:

on:
  push:
 
jobs:

  use-quickstart:
    needs: build-custom
    runs-on: ubuntu-latest
    steps:
    - uses: stellar/quickstart@main
      with:
        tag: latest

To run a quickstart that has been built with a custom configuration, do this:

on:
  push:
 
jobs:

  build-custom:
    uses: stellar/quickstart/.github/workflows/build.yml@main
    with:
      images: |
        [
          {
            "tag": "custom",
            "config": {
              "protocol_version_default": 23
            },
            "deps": [
              { "name": "xdr", "repo": "stellar/rs-stellar-xdr", "ref": "v23.0.0" },
              { "name": "core", "repo": "stellar/stellar-core", "ref": "v23.0.1", "options": { "configure_flags": "--disable-tests" } },
              { "name": "rpc", "repo": "stellar/stellar-rpc", "ref": "v23.0.1" },
              { "name": "horizon", "repo": "stellar/go", "ref": "horizon-v23.0.0" },
              { "name": "friendbot", "repo": "stellar/go", "ref": "horizon-v23.0.0" },
              { "name": "lab", "repo": "stellar/laboratory", "ref": "main" }
            ],
            "additional-tests": []
          }
        ]
      archs: '["amd64"]'

  use-custom:
    needs: build-custom
    runs-on: ubuntu-latest
    steps:
    - uses: stellar/quickstart@main
      with:
        artifact: image-quickstart-custom-amd64.tar
        tag: custom-amd64

See an example of that working here:

• https://github.com/leighmcculloch/test-stellar-quickstart-workflow-call/blob/main/.github/workflows/workflow-call-test.yml
• https://github.com/leighmcculloch/test-stellar-quickstart-workflow-call/actions/runs/18332505803/job/52210559423

And example of using it with the stellar/stellar-core repo:

• https://github.com/leighmcculloch/stellar--stellar-core/actions/runs/18336142994/workflow

Why

This work started out to make some small improvements to the build pipeline, because the pipeline defined versions in multiple places and people were understandably frequently forgetting to update the versions everywhere they needed to be.

The work then extended to improve the scalability to reduce the time spend rebuilding the same software in parallel when the same software was used for multiple images during the same build.

During the work it occurred to me that if structured appropriately the rewrite could present the pipeline in a way to also support nightly scheduled runs and even building from other repos.

The bonus features were included because it's always been an annoying shortcoming of quickstart that it can build anything, but only for itself. And it would be helpful if you could use quickstarts pipeline to build nightly builds, or any other custom builds of all the software that ends up in quickstart.

In the past we've built secondary pipelines in things like system-test, rather than leveraging the caching and build process that exists here in quickstart. With this change, repos like system-test will be able to stay focused on their test cases, and less on trying to replicate the build process.

Close #754
Close #651
Close #705

cc @stellar/devx @stellar/platform-eng @sisuresh