Add initial FOCIL implementation #30914

jihoonsong · 2024-12-14T14:51:20Z

This PR implements initial FOCIL specifications. For more information, please refer to EIP-7805 and this PR.

beacon/engine/types.go

ensi321

Pointing out Engine API has incorrect type for inclusion list. Current implementation expects base64 strings whereas the convention and the specs demand hex strings.

I suggest either define inclusion list type and its marshal type in beacon/engine/types.go and let gencodec handles the conversion similar to how ExecutableData is handled. Or manually marshal/unmarshal like how convertRequests handles incoming executionRequests in NewPayloadV4

eth/catalyst/api.go

beacon/engine/types.go

jihoonsong · 2025-02-07T14:08:55Z

I've addressed comments. Thank you very much for spotting out the issue and suggesting solutions! @ensi321

beacon/engine/types.go

eth/catalyst/api.go

jihoonsong · 2025-05-19T05:19:38Z

@jwasinger Applied your suggestion and rebased, thanks!

jwasinger · 2025-05-19T05:53:40Z

eth/catalyst/api.go

-		case !api.checkFork(params.Timestamp, forks.Cancun, forks.Prague):
-			return engine.STATUS_INVALID, unsupportedForkErr("fcuV3 must only be called for cancun or prague payloads")
+		if params.Withdrawals == nil {
+			return engine.STATUS_INVALID, engine.InvalidPayloadAttributes.With(errors.New("missing withdrawals"))


The latest rebase removed the refactors that were recently merged into this file.

Current implementation expects fcuv3 be called in eip7805 fork as well, which will be absorbed into some canonical fork in future.

The review wasn't about this line in particular, sorry for the confusion. We refactored a lot of if/else if/else if/... into switch statements.

Oh right, the last commit reverts all of those changes without causing any conflict. My bad, let me reflect the refactors.

By the way, it seems the refactor can be extended a bit. e.g, replacing engine.PayloadStatusV1{Status: engine.INVALID} with invalidStatus, and declaring a common variable for engine.StatelessPayloadStatusV1{Status: engine.INVALID}. Also, some comments are off. L609 should be return invalidStatus, unsupportedForkErr("newPayloadV4 must only be called for prague payloads"). Same with L686 and L763. (Line numbers are based on the latest master branch)

eth/catalyst/api.go

miner/payload_building.go

…on list

… on eip7805 fork

jwasinger · 2025-05-21T05:49:46Z

miner/worker.go

+		pendingInclusionListTxs = append(pendingInclusionListTxs, tx)
+	}
+
+	// Append any valid inclusion list transactions at the end of the block.


We should sort IL+pool transactions by how many fees they pay and use this to prioritize inclusion. It's possible there could be high-paying txs in the IL that aren't in our pool.

Absolutely. Current implementation is a naive one, prioritized easier implementation and testing. Do you think we need to adapt the inclusion strategy in this PR?

It's not absolutely vital, but we will want to make the change before this goes into production.

Is there anything else that you omitted in this PR to keep the diff easier to read/test?

Nothing comes to my mind at the moment. Whether to use engine_updatePayloadWithInclusionListV1 or engine_forkchoiceUpdatedV4 could be changed in future as it's not decided yet.

I've come up with another one. It takes a naive approach when building an IL, too. It doesn't use any heuristic algorithm such as taking the time spent in the mempool into account, and so on.

Like I wrote on the call: because it's unlikely that we will simultaneously be an IL committee member and the proposer for the same block, maybe we can afford to have a more altruistic default strategy than just building the most profitable IL.

and also to build the most profitable IL, we would have to do full transaction execution here. and even then, it's still speculative because we are not building the IL on top of the state that it will be applied.

eth/catalyst/api.go

MariusVanDerWijden · 2025-05-21T08:16:22Z

eth/catalyst/api.go

@@ -769,7 +865,7 @@ func (api *ConsensusAPI) ExecuteStatelessPayloadV4(params engine.ExecutableData,
 	return api.executeStatelessPayload(params, versionedHashes, beaconRoot, requests, opaqueWitness)
 }

-func (api *ConsensusAPI) newPayload(params engine.ExecutableData, versionedHashes []common.Hash, beaconRoot *common.Hash, requests [][]byte, witness bool) (engine.PayloadStatusV1, error) {
+func (api *ConsensusAPI) newPayload(params engine.ExecutableData, versionedHashes []common.Hash, beaconRoot *common.Hash, requests [][]byte, inclusionList engine.InclusionList, witness bool) (engine.PayloadStatusV1, error) {


The parameters here become quite clunky. Maybe we should create a struct here to pass the data, makes it easier to add fields in the future

Fair point. But until we get into this implementation seriously, I'd like to keep it simple and naive as master can be refactored. I believe applying these changes into master when the time comes would be cleaner.

eth/catalyst/queue.go

MariusVanDerWijden · 2025-05-21T08:18:40Z

miner/inclusion_list_building.go

+		return nil, err
+	}
+
+	if err := miner.fillTransactions(nil, env); err != nil {


We will fill with as many transactions as possible, but only the first 8 will actually be included in the inclusion list, right? Maybe we should limit the transactions to 8 in that case to not spend too much extra time on this

The IL size of 8kb is the limit per proposer in the committee. The aggregated IL that's passed to the EL for block production can be 8*16 kb max.

…usion list

…yload`

jihoonsong · 2025-05-29T10:00:12Z

At the call with Geth team, there was feedback that eth/catalyst/api.go is not the right place to put validateInclusionListConstraints.

I haven't given much thought about where is the best location from the Geth codebase perspective, but from the spec perspective, the IL constraints should be checked after executing all transactions and before processing withdrawal requests. This is to prevent such situations that some withdrawal address become availble to cover some IL transaction that they couldn't. That missing IL transaction can reorg the block, which it shouldn't.

In current implementation, it would be between L106-L107 of core/state_processor.go.

As this would change the core part of Geth, I would wait until I can get more feedback from the Geth team before proceeding.

jihoonsong requested review from MariusVanDerWijden, holiman, fjl, rjl493456442, lightclient and jwasinger as code owners December 14, 2024 14:51

s1na self-assigned this Dec 16, 2024

jihoonsong force-pushed the focil branch from 1d9b8a8 to 7ba7b57 Compare December 30, 2024 07:47

lightclient reviewed Jan 7, 2025

View reviewed changes

beacon/engine/types.go Outdated Show resolved Hide resolved

jihoonsong force-pushed the focil branch 3 times, most recently from 33efbee to 78f8ad9 Compare January 9, 2025 08:46

jihoonsong force-pushed the focil branch 2 times, most recently from 9f5db6f to bbdaa45 Compare January 19, 2025 05:36

jihoonsong force-pushed the focil branch from bbdaa45 to 93a5a14 Compare January 22, 2025 06:22

jihoonsong force-pushed the focil branch from 93a5a14 to 1b47c09 Compare February 1, 2025 19:33

ensi321 reviewed Feb 7, 2025

View reviewed changes

eth/catalyst/api.go Outdated Show resolved Hide resolved

beacon/engine/types.go Show resolved Hide resolved

jihoonsong force-pushed the focil branch from 1b47c09 to 4ae0066 Compare February 7, 2025 13:54

s1na reviewed Feb 21, 2025

View reviewed changes

beacon/engine/types.go Outdated Show resolved Hide resolved

eth/catalyst/api.go Outdated Show resolved Hide resolved

jihoonsong force-pushed the focil branch 2 times, most recently from eb46040 to d5c724b Compare February 24, 2025 16:26

Marchhill reviewed Feb 24, 2025

View reviewed changes

eth/catalyst/api.go Outdated Show resolved Hide resolved

jihoonsong force-pushed the focil branch from d5c724b to cb1773c Compare February 25, 2025 07:07

jihoonsong force-pushed the focil branch 3 times, most recently from 9e1f361 to 29b805d Compare March 21, 2025 14:11

jihoonsong force-pushed the focil branch from 47a07e5 to 7394683 Compare April 8, 2025 12:55

jihoonsong requested review from karalabe and gballet as code owners April 8, 2025 12:55

jwasinger reviewed May 19, 2025

View reviewed changes

eth/catalyst/api.go Show resolved Hide resolved

jihoonsong force-pushed the focil branch from 266f63c to caef2da Compare May 19, 2025 08:35

jwasinger reviewed May 19, 2025

View reviewed changes

miner/payload_building.go Show resolved Hide resolved

jihoonsong added 8 commits May 21, 2025 14:04

beacon/engine: add InclusionList type

a068aa2

beacon/engine: add helpers for the conversion between txs and inclusi…

f984828

…on list

eth/catalyst, miner: add GetInclusionListV1

0b46f10

eth/catalyst: cache recently generated inclusion lists

89896f8

beacon/engine: add INVALID_INCLUSION_LIST to payload status

69c51b0

consensus/misc/eip4844, params: support eip7805 fork

4026a0f

eth/catalyst: make forkchoiceUpdatedV3 and newPayloadV4 available…

90d59b8

… on eip7805 fork

eth/catalyst: add engine_newPayloadV5

bbcade2

jihoonsong force-pushed the focil branch from caef2da to 928b123 Compare May 21, 2025 05:05

jwasinger reviewed May 21, 2025

View reviewed changes

MariusVanDerWijden reviewed May 21, 2025

View reviewed changes

eth/catalyst/api.go Outdated Show resolved Hide resolved

MariusVanDerWijden reviewed May 21, 2025

View reviewed changes

eth/catalyst/queue.go Outdated Show resolved Hide resolved

MariusVanDerWijden reviewed May 21, 2025

View reviewed changes

jihoonsong added 9 commits May 21, 2025 18:38

eth/catalyst: verify if a block satisfies the inclusion list constraints

fc8588f

eth/catalyst: add an unit test for verifying new payload against incl…

d90735f

…usion list

miner: add inclusionList to Miner.generateParams

bc38736

miner: add inclusionList to Miner.environment

f635a7e

miner: include inclusion list transactions when building a payload

236aa69

miner: add a public method to notify the inclusion list to payload

a5701a6

eth/catalyst: add peek method to payloadQueue to return `Miner.Pa…

d3bfb8d

…yload`

eth/catalyst: add engine_updatePayloadWithInclusionListV1

c5b634e

eth/catalyst: add an unit test for updating payload with inclusion list

9f03028

jihoonsong force-pushed the focil branch from 928b123 to 9f03028 Compare May 21, 2025 09:40

Add initial FOCIL implementation #30914

Are you sure you want to change the base?

Add initial FOCIL implementation #30914

Conversation

jihoonsong commented Dec 14, 2024

Uh oh!

Uh oh!

ensi321 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jihoonsong commented Feb 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jihoonsong commented May 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jihoonsong May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jihoonsong commented May 29, 2025

Uh oh!

Uh oh!

jihoonsong May 19, 2025 •

edited

Loading