Skip to content
1 change: 1 addition & 0 deletions cmd/ethrex_replay/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -58,3 +58,4 @@ l2 = [
profiling = ["ethrex-prover/profiling"]
ci = []
jemalloc = ["dep:tikv-jemallocator"]
revm = ["ethrex-vm/revm"]
2 changes: 1 addition & 1 deletion docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@
- [Ethrex L2 as local development mode](./developers/l2/dev-mode.md)
- [Debugging solidity with ethrex](./vm/levm/debug.md)
- [Re-execute Ethereum with ethrex](./ethrex_replay/ethrex_replay.md)
- [Profiling zkvm execution with ethrex replay](./ethrex_replay/profiling.md)
- [FAQ](./ethrex_replay/faq.md)
- [CLI reference](./CLI.md)
- [Troubleshooting]()

Expand Down
71 changes: 37 additions & 34 deletions docs/ethrex_replay/ethrex_replay.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,38 @@
# ethrex-replay

A tool for executing and proving Ethereum blocks, transactions, and L2 batches — inspired by [starknet-replay](https://github.com/lambdaclass/starknet-replay).
Currently ethrex replay only works against ethrex nodes with the `debug_executionWitness` RPC endpoint.

## Status
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that this was a very big table and now both execution and proving works. So I just left the little table below this that has client compatibility, I think that's enough given that we already support every network.


| Node | Network | `ethrex-replay execute block` | Additional Comments | `ethrex-replay prove block` | Additional Comments |
| ---------- | ------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------- |
| reth | hoodi | ✅ | Most of the recent blocks can be executed with or without SP1 and proved with SP1 | ✅ | Every block that is successfully executed with SP1, can also be successfully proved. |
| reth | sepolia | ✅ | Most of the recent blocks can be executed with or without SP1 and proved with SP1 | ✅ | SP1 panicked in all attempts to prove blocks. |
| reth | mainnet | ✅ | Works reliably. Execution with SP1 works in most of the blocks, but with SP1 it works with ~8/10 success rate | ✅ | Works reliably. Execution with SP1 works in most of the blocks, but with SP1 it works with ~8/10 success rate |
| geth | hoodi | ✅ | - | 🔜 | - |
| geth | sepolia | 🔜 | - | 🔜 | - |
| geth | mainnet | 🔜 | - | 🔜 | - |
| nethermind | hoodi | 🏗️ | Fails sometimes. | 🔜 | - |
| nethermind | sepolia | 🔜 | - | 🔜 | - |
| nethermind | mainnet | 🔜 | - | 🔜 | - |
| ethrex | hoodi | ✅ | - | ✅ | - |
| ethrex | sepolia | ✅ | - | ✅ | - |
| ethrex | mainnet | ✅ | Works reliably. Execution with SP1 works in most of the blocks, but with SP1 it works with ~8/10 success rate | ✅ | Works reliably. Execution with SP1 works in most of the blocks, but with SP1 it works with ~8/10 success rate |
| erigon | hoodi | 🔜 | - | 🔜 | - |
| erigon | sepolia | 🔜 | - | 🔜 | - |
| erigon | mainnet | 🔜 | - | 🔜 | - |

## Supported Networks

- Mainnet
- Hoodi
- Sepolia

## Client Compatibility

| Client | `ethrex-replay execute block` | `ethrex-replay prove block` |
| ---------- | ----------------------------- | --------------------------- |
| reth | ✅ | ✅ |
| geth | ✅ | ✅ |
| nethermind | 🏗️ | 🏗️ |
| ethrex | ✅ | ✅ |
| erigon | 🔜 | 🔜 |
| Client | `ethrex-replay block` | notes |
| ---------- | ----------------------------- | ------------------------------------------------ |
| reth | ✅ | `debug_executionWitness` |
| geth | ✅ | `eth_getProof` |
| nethermind | ✅ | `eth_getProof` (hoodi is unstable) |
| ethrex | ✅ | `debug_executionWitness` |
| erigon | ❌ | V3 supports `eth_getProof` only for latest block |
| besu | ❌ | Doesn't return proof for non-existing accounts |

We support any other client that responds correctly to `eth_getProof` or `debug_executionWitness` endpoints.

Execution of some particular blocks with the `eth_getProof` method won't work with zkVMs. But without using these it should work for any block. Read more about this in [FAQ](./faq.md). Also, when running against a **full node** using `eth_getProof` if for some reason execution were to take longer than 25 minutes it would probably fail because the node may have pruned it's state (128 blocks * 12 seconds = 25,6 min), normally it doesn't take that much but be wary of that.

Note: For some reason Nethermind with hoodi nodes (up to v1.33 at least) doesn't work properly for some blocks, we get gas used mismatch. The witness generated with this method has fewer trie nodes than the witness generated with geth nodes, we don't know why. This hasn't happened in mainnet or sepolia yet.


## Getting Started

### Dependencies

These dependencies are optional, install them only if you want to run with the features `risc0` or `sp1` respectively.
Make sure to use the correct versions of these.

#### [RISC0](https://dev.risczero.com/api/zkvm/install)

```sh
Expand All @@ -46,7 +42,7 @@ rzup install risc0-groth16
rzup install rust
```

#### [SP1](https://docs.succinct.xyz/docs/sp1/introduction)
#### [SP1](https://docs.succinct.xyz/docs/sp1/getting-started/install)

```sh
curl -L https://sp1up.succinct.xyz | bash
Expand Down Expand Up @@ -117,6 +113,9 @@ cargo r -r -p ethrex-replay --features risc0 -- <COMMAND> [ARGS]
## RISC0 backend + GPU
cargo r -r -p ethrex-replay --features risc0,gpu -- <COMMAND> [ARGS]

## REVM as EVM Backend
cargo r -r -p ethrex-replay --features revm -- <COMMAND> [ARGS]

# L2 replay

## Vanilla execution (no prover backend)
Expand Down Expand Up @@ -147,6 +146,7 @@ The following table lists the available features for `ethrex-replay`. To enable
| `l2` | Enables L2 batch execution and proving (can be combined with SP1 or RISC0 and GPU features, e.g. `sp1,l2,gpu`, `risc0,l2,gpu`, `sp1,l2`, `risc0,l2`) |
| `jemalloc` | Use jemalloc as the global allocator. This is useful to combine with tools like Bytehound and Heaptrack for memory profiling |
| `profiling` | Useful to run with tools like Samply. |
| `revm` | For running with REVM as backend instead of LEVM. Note that it will only work for L1 and without SP1 or RISC0 |

---

Expand Down Expand Up @@ -227,6 +227,11 @@ ethrex-replay block-composition --start-block <START_BLOCK> --end-block <END_BLO

### Run Samply

We recommend building in `release-with-debug` mode so that the flamegraph is the most accurate.
```bash
cargo build -p ethrex-replay --profile release-with-debug --features <FEATURES>
```

#### On zkVMs

> [!IMPORTANT]
Expand All @@ -235,16 +240,13 @@ ethrex-replay block-composition --start-block <START_BLOCK> --end-block <END_BLO
> 2. The `TRACE_SAMPLE_RATE` environment variable controls the sampling rate (in milliseconds). Adjust it according to your needs.

```
TRACE_FILE=output.json TRACE_SAMPLE_RATE=1000 ethrex-replay <COMMAND> [ARGS]
TRACE_FILE=output.json TRACE_SAMPLE_RATE=1000 target/release-with-debug/ethrex-replay <COMMAND> [ARGS]
```

#### Execution without zkVMs

We recommend building in release-with-debug mode so that the flamegraph is the most accurate.

```bash
cargo build -p ethrex-replay --profile release-with-debug
samply record target/release-with-debug/ethrex-replay <COMMAND> [ARGS]
samply record target/release-with-debug/ethrex-replay <COMMAND> --no-zkvm [OTHER_ARGS]
```

### Run Bytehound
Expand All @@ -265,6 +267,7 @@ LD_PRELOAD=/path/to/bytehound/preload/target/release/libbytehound.so:/path/to/li
>
> 1. The following requires [Jemalloc](https://github.com/jemalloc/jemalloc) and [Heaptrack](https://github.com/KDE/heaptrack) to be installed.
> 2. The `ethrex-replay` binary must be built with the `jemalloc` feature enabled.
> 3. Note that Heaptrack is a **Linux** profiler, so it won't work natively on macOS.

```
LD_PRELOAD=/path/to/libjemalloc.so heaptrack ethrex-replay <COMMAND> [ARGS]
Expand Down
28 changes: 28 additions & 0 deletions docs/ethrex_replay/faq.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
## FAQ

### What's the difference between `eth_getProof` and `debug_executionWitness`?

`eth_getProof` gets the proof for a particular account and the chosen storage slots.
`debug_executionWitness` gets the whole execution witness necessary to execute a block in a stateless manner.

The former endpoint is implemented by all execution clients and you can even find it in RPC Providers like Alchemy, the latter is only implemented by some execution clients and you can't find it in RPC Providers.

When wanting to execute a historical block we tend to use the `eth_getProof` method with an RPC Provider because it will be the most reliable, other way is using it against a Hash-Based Archive Node but this would be too heavy to host ourselves (20TB at least). This method is slow because it performs many requests but it's very flexible.

If instead we want to execute a recent block we use it against synced ethrex or reth nodes that expose the `debug_executionWitness` endpoint, this way retrieval of data will be instant and it will be way faster than the other method, because it won't be doing thousands of RPC requests, just one.

### Why stateless execution of some blocks doesn't work with `eth_getProof`

With this method of execution we get the proof of all the accounts and storage slots accessed during execution, but the problem arises when we want to delete a node from the Merkle Patricia Trie (MPT) when applying the account updates of the block. This is for a particular case in which a tree restructuring happens and we have a missing node that wasn't accessed but we need to know in order to restructure the trie.

The problem can be explained with a simple example: a Branch node has 2 child nodes and only one was accessed and removed, this branch node should stop existing because they shouldn't have only **one** child. It will be either replaced by a leaf node or by an extension node, this depends on its child.

This problem is wonderfully explained in [zkpig docs](https://github.com/kkrt-labs/zk-pig/blob/main/docs/modified-mpt.md), they also have a very good intro to the MPT.
Here they mention two different solutions that we have to implement in order to fix this. The first one works when the missing node is a Leaf or Extension and the second one works then the missing node is a Branch.

In our code we only applied the first solution by injecting all possible nodes to the execution witness that we build when using `eth_getProof`, that's why the witness when using this method will be larger than the witness obtained with `debug_executionWitness`.

We didn't apply the second change because it needs a change to the MPT that we don't want in our code. However we were able to solve it for execution without using a zkVM by injecting some "fake nodes" to the trie just before execution that have the expected hash but their RLP content doesn't match to it. This way we can "trick" the Trie into thinking that it has the branch nodes when in fact, it doesn't.



70 changes: 0 additions & 70 deletions docs/ethrex_replay/profiling.md
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one was outdated

This file was deleted.