feat: Add `CertificateChainSynchronizer` and make follower aggregators start their chain by synchronising with their leader #2634

Alenar · 2025-07-08T15:00:12Z

Content

This PR add a certificate chain synchronization mechanism allowing follower Aggregators to avoid creating genesis certificate by, instead, synchronizing with their leader.

The synchronizer behaves as follows:

It check its force parameter (that allow the state machine to force a synchronization based on a condition, today it forces the sync if the follower chain is invalid), else the sync is done if either:
- there's no chain yet on the follower aggregator
- the latest remote genesis certificate has changed (it's different than the stored latest genesis certificate)
If any of the previous is true it proceed by fetching the latest certificate from the remote/leader aggregator
Using a CertificateVerifier wired to the remote aggregator, it fetches the remote certificate chains and validate it at the same time
- Only the first certificate of each epoch is kept
It stores the fetched certificate in the database using a insert or replace to keep the data up to date
it finish by creating a certified OpenMessage for MithrilStakeDistribution in the database so the aggregator won't create a certificate for this signed entity

Main changes

Aggregator:
- New service: CertificateChainSynchronizer
  - Define several traits:
    - CertificateChainSynchronizer: main trait, define the service api
    - RemoteCertificateRetriever: define how a synchronizer get the latest certificate of the remote (to use it has the starting point of the synchronization) and the latest genesis of the remote (to check if it have changed, triggering a sync)
    - SynchronizedCertificateStorer: define how retrieved certificates are stored and how to retrieve the latest stored genesis certificate of the aggregator
    - OpenMessageStorer: define how the open message created at the end of the process is stored
  - Two implementations:
    - MithrilCertificateChainSynchroniserNoop: allow to construct a synchronizer in the leader aggregator dependency builder
    - MithrilCertificateChainSynchronizer: main implementation
- state machine and runner: call synchroniser when transitioning from idle to ready if the aggregator is a follower.
- database, two new queries: InsertOrReplaceCertificateRecordQuery and InsertOrReplaceOpenMessageQuery
- AggregatorHTTPClient: support three new queries, latest_certificates_list, certificates_details, and latest_genesis_certificate
- create_certificate_follower integration tests: updated to check the synchronizer instead of spawning a genesis
  certificate on the follower (this simplified the scenario since one less epoch is needed).
End to end:
- only the leader aggregator is started at the infrastructure creation
- start the follower only after the completion of a "bootstrap" step on the leazder the goes until the creation of the genesis certificate and the delegation of some stakes to the pools (since it was done only on the "leader")

Additional changes

fix(aggregator): giving a aggregator_url with a trailing slash to AggregatorHTTPClient would makes all its queries fails
refactor(aggregator): Follow DI pattern for the relation of the MithrilSignerRegistrationFollower and the AggregatorHTTPClient by moving and renaming the AggregatorClient trait to it's api defined traits.
refactor(aggregator): Store the AggregatorHTTPClient directly as its concrete type in the dependencies builder, this storing it multiple times, one for each of the traits it implement.
refactor(aggregator-integration tests): replace TestHttpServer with axum-test as the backend for the leader aggregator http server.
refactor(aggregator-integration tests): extract LeaderAggregatorHttpServer to a dedicated module
feat(common-CertificateChainBuilder): add latest_certificate to the fixture

Pre-submit checklist

Branch
- Tests are provided (if possible)
- Crates versions are updated (if relevant)
- CHANGELOG file is updated (if relevant)
- Commit sequence broadly makes sense
- Key commits have useful messages
PR
- All check jobs of the CI have succeeded
- Self-reviewed the diff
- Useful pull request description
- Reviewer requested
Documentation
- No new TODOs introduced

Issue(s)

Relates to #2534

Copilot

Pull Request Overview

This PR implements a certificate chain synchronization mechanism so that follower aggregators can sync their chain from a leader rather than creating a genesis certificate.

Introduces a CertificateChainSynchronizer service (and a no-op variant for leaders)
Refactors infrastructure startup to prepare and register leader/follower aggregators and serve them in sequence
Updates the runtime state machine to validate and (if needed) synchronize the follower’s certificate chain before proceeding

Reviewed Changes

Copilot reviewed 40 out of 41 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
mithril-test-lab/mithril-end-to-end/src/run_only.rs	Simplify run flow to bootstrap the leader then serve followers
mithril-test-lab/mithril-end-to-end/src/mithril/infrastructure.rs	Refactor aggregator startup to prepare leader/follower and register era
mithril-aggregator/src/services/certificate_chain_synchroniser/synchroniser_service.rs	Add certificate chain synchronization logic
mithril-aggregator/src/services/aggregator_client.rs	Extend HTTP client to fetch certificate lists and details
mithril-aggregator/src/runtime/state_machine.rs	Validate and synchronize follower certificate chain in the runner

Comments suppressed due to low confidence (1)

mithril-test-lab/mithril-end-to-end/src/mithril/infrastructure.rs:318

The variable leader_aggregator_endpoint is not defined in this scope; you probably meant to use aggregator_endpoints[0] or introduce leader_aggregator_endpoint before this line.

                aggregator_endpoint: &leader_aggregator_endpoint,

mithril-test-lab/mithril-end-to-end/src/run_only.rs

mithril-test-lab/mithril-end-to-end/src/end_to_end_spec.rs

github-actions · 2025-07-08T15:07:45Z

Test Results

4 files ± 0 154 suites ±0 22m 53s ⏱️ + 1m 44s
2 118 tests +39 2 118 ✅ +39 0 💤 ±0 0 ❌ ±0
6 466 runs +80 6 466 ✅ +80 0 💤 ±0 0 ❌ ±0

Results for commit 11c8aec. ± Comparison against base commit b7a272c.

♻️ This comment has been updated with latest results.

if the leader aggregator url was specified with a trailing slash, request to it would fails because joined url would have two slashes, i.e. `http://aggregator//route`. This align with the behavior of the aggregator client of the mithril-client lib.

… signer registration follower several benefits: - cleaner cut between signer registration business layer and its infrastructure - allow to use the concrete `AggregatorHTTPClient` type in dependency injection, allowing to implement multiple traits on it and still pass it to the multiple depenedencies that use that trait

…o a module In integration tests

…erver Instead of warp based `mithril-test-http-server`, as this allow to write simpler, more readable, code.

…in retrieval tests

Nothing is stored yet, this will come afterward

…onizer`

…hroniser

This allow implementor to use transactions if needed.

…Client`

This is critical since rows are returned from the database by insertion order reversed.

…poch plus the genesis.

… entity only if not synchronized Has synchronised certificate are created without a signed entity, previouysly it was only the case for genesis certificates.

mithril-aggregator/src/services/certificate_chain_synchroniser/synchroniser_service.rs

jpraynaud

LGTM 👍

I left some minor comments.

mithril-aggregator/tests/create_certificate_follower.rs

mithril-test-lab/mithril-end-to-end/src/end_to_end_spec.rs

mithril-aggregator/src/services/certificate_chain_synchroniser/synchroniser_service.rs

dlachaume

LGTM 🚀

mithril-aggregator/src/services/certificate_chain_synchroniser/synchroniser_service.rs

mithril-aggregator/src/services/aggregator_client.rs

turmelclem

LGTM

- fix: synchronizer create the open messages based on the first certificate of the last epoch instead of the latest certificate - move `MockCertificateVerifier` definition to `tools::mocks` - fix spelling issues

… after sync This ensure that the synchronizer create an open message for MSD and that the state machine then proceed to the next signed entity type.

CHANGELOG.md

* mithril-aggregator from `0.7.72` to `0.7.73` * mithril-common from `0.6.7` to `0.6.8` * mithril-end-to-end from `0.4.95` to `0.4.96`

Alenar requested review from jpraynaud, dlachaume, turmelclem and Copilot July 8, 2025 15:00

Alenar self-assigned this Jul 8, 2025

Alenar added the feature 🚀 New implemented feature label Jul 8, 2025

Copilot AI reviewed Jul 8, 2025

View reviewed changes

mithril-test-lab/mithril-end-to-end/src/run_only.rs Outdated Show resolved Hide resolved

mithril-test-lab/mithril-end-to-end/src/end_to_end_spec.rs Outdated Show resolved Hide resolved

Alenar force-pushed the djo/2534/aggregator-sync-master_follower branch from 72b7d34 to a46e42b Compare July 8, 2025 15:57

Alenar added 21 commits July 8, 2025 17:57

refactor(aggregator-test): extract leader follower test http server t…

7163579

…o a module In integration tests

refactor(aggregator-test): use axum test for leader aggregator http s…

a22eedc

…erver Instead of warp based `mithril-test-http-server`, as this allow to write simpler, more readable, code.

feat(aggregator): scaffold CertificateChainSynchronizer

f31bbb8

feat(aggregator): impl logic that check if sync is needed

d97c731

test(aggregator): rework synchroniser test tools & prepare remote cha…

053ba8c

…in retrieval tests

feat(aggregator): implement logic to retrieve/validate remote chain

34b578b

Nothing is stored yet, this will come afterward

feat(aggregator): implement logic to store retrieved chain

ee585f7

feat(aggregator): add logs & error context to `CertificateChainSynchr…

8be4f82

…onizer`

refactor(aggregator): add a enum to make sync reason explicit in sync…

4ec7378

…hroniser

refactor(aggregator): make storage of sync certificates works on a batch

f48f558

This allow implementor to use transactions if needed.

test(common): add latest_certificate to CertificateChainFixture

7ccd88c

test(aggregator): add 'e2e' tests to synchroniser

6a4c9c0

feat(aggregator): add RemoteCertificateRetriever to `AggregatorHTTP…

b855ab3

…Client`

feat(aggregator): ensure synced data is stored from genesis to latest

d63b82b

This is critical since rows are returned from the database by insertion order reversed.

feat(aggregator): make synchroniser add an open message at process end

70dcce9

feat(aggregator): make synchroniser only persist first cert of each e…

c45076c

…poch plus the genesis.

feat(aggregator): implement persistence of synchronised certificates

55d7d32

feat(aggregator): implement persistence of openmessage created post sync

8931cb8

feat(aggregator): add follower sync in state machine + wire synchroniser

930ee6e

Alenar added 3 commits July 8, 2025 17:57

test(aggregator): update tooling to check cert have associated signed…

c5fa040

… entity only if not synchronized Has synchronised certificate are created without a signed entity, previouysly it was only the case for genesis certificates.

test(aggregator): update create_certificate_follower integration test

3d4734d

test: update e2e to follower auto-sync

9bebed2

Alenar force-pushed the djo/2534/aggregator-sync-master_follower branch from a46e42b to 9bebed2 Compare July 8, 2025 15:57

Alenar commented Jul 8, 2025

View reviewed changes

mithril-aggregator/src/services/certificate_chain_synchroniser/synchroniser_service.rs Outdated Show resolved Hide resolved

mithril-aggregator/src/services/certificate_chain_synchroniser/synchroniser_service.rs Outdated Show resolved Hide resolved

Alenar temporarily deployed to testing-preview July 8, 2025 16:32 — with GitHub Actions Inactive

jpraynaud approved these changes Jul 8, 2025

View reviewed changes

dlachaume approved these changes Jul 9, 2025

View reviewed changes

mithril-aggregator/src/services/certificate_chain_synchroniser/synchroniser_service.rs Outdated Show resolved Hide resolved

mithril-aggregator/src/services/aggregator_client.rs Outdated Show resolved Hide resolved

turmelclem approved these changes Jul 9, 2025

View reviewed changes

Alenar added 2 commits July 9, 2025 18:07

fix: address PR reviews comments

1796fdb

- fix: synchronizer create the open messages based on the first certificate of the last epoch instead of the latest certificate - move `MockCertificateVerifier` definition to `tools::mocks` - fix spelling issues

test(aggregato): in create_certificate_follower check open messages…

a54c3be

… after sync This ensure that the synchronizer create an open message for MSD and that the state machine then proceed to the next signed entity type.

Alenar force-pushed the djo/2534/aggregator-sync-master_follower branch from e05637f to d349f0b Compare July 9, 2025 16:08

dlachaume approved these changes Jul 10, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Alenar temporarily deployed to testing-preview July 10, 2025 07:50 — with GitHub Actions Inactive

Alenar added 2 commits July 10, 2025 10:15

chore: update changelog

a9d24d8

chore: upgrade crate versions

11c8aec

* mithril-aggregator from `0.7.72` to `0.7.73` * mithril-common from `0.6.7` to `0.6.8` * mithril-end-to-end from `0.4.95` to `0.4.96`

Alenar force-pushed the djo/2534/aggregator-sync-master_follower branch from d349f0b to 11c8aec Compare July 10, 2025 08:15

Alenar temporarily deployed to testing-preview July 10, 2025 08:29 — with GitHub Actions Inactive

Alenar merged commit e52b52a into main Jul 10, 2025
70 of 71 checks passed

Alenar deleted the djo/2534/aggregator-sync-master_follower branch July 10, 2025 08:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add `CertificateChainSynchronizer` and make follower aggregators start their chain by synchronising with their leader #2634

feat: Add `CertificateChainSynchronizer` and make follower aggregators start their chain by synchronising with their leader #2634

Uh oh!

Alenar commented Jul 8, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

jpraynaud left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dlachaume left a comment

Uh oh!

Uh oh!

Uh oh!

turmelclem left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat: Add CertificateChainSynchronizer and make follower aggregators start their chain by synchronising with their leader #2634

feat: Add CertificateChainSynchronizer and make follower aggregators start their chain by synchronising with their leader #2634

Uh oh!

Conversation

Alenar commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content

Main changes

Additional changes

Pre-submit checklist

Issue(s)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Uh oh!

Uh oh!

jpraynaud left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dlachaume left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

turmelclem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat: Add `CertificateChainSynchronizer` and make follower aggregators start their chain by synchronising with their leader #2634

feat: Add `CertificateChainSynchronizer` and make follower aggregators start their chain by synchronising with their leader #2634

Alenar commented Jul 8, 2025 •

edited

Loading

github-actions bot commented Jul 8, 2025 •

edited

Loading

jpraynaud left a comment •

edited

Loading