[POC] new database interface #4623

damiramirez · 2025-09-23T20:06:21Z

Warning

This is a large refactor in progress. The API and implementation are subject to change

Motivation

Our current StoreEngine interface has grown into a God Object with over 50 methods, making it difficult to add new database backends. Business logic is scattered across every database implementation, leading to code duplication and maintenance overhead.

This PR addresses these issues by introducing a minimal, focused interface that makes adding new databases to ethrex trivial.

Also removes libmdbx from ethrex

Description

This PR implements a fundamental redesign of the storage layer using a three-tier architecture: Storage → Storage Engine → Storage Backend → Database Implementation

Store
- Implements Store wrapping StoreEngine
- High-level blockchain operations
StorageEngine
- Handles complex operations
- Manages data serialization/deserialization using RLP encoding
StorageBackend
- Defines the contract for database operations with transactions
- Supports basic DB operations

New storage structure

crates/storage
├── api.rs
├── backend
│   ├── in_memory.rs
│   ├── mod.rs
│   └── rocksdb.rs
├── Cargo.toml
├── clippy.toml
├── engine.rs
├── error.rs
├── lib.rs
├── rlp.rs
├── store.rs
├── trie.rs
└── utils.rs

The new StorageBackend trait provides a thin abstraction over database operations:

// This can be used when we have a database that supports dupsort tables or other options
// This struct exists but I'm not using it for now
#[derive(Debug, Clone)]
pub struct TableOptions {
    pub dupsort: bool,
}

pub trait StorageBackend: Debug + Send + Sync {
    fn open(path: impl AsRef<Path>) -> Result<Self, StoreError>
    // Check if needed
    fn create_table(&self, name: &str, options: TableOptions) -> Result<(), StoreError>;
    fn clear_table(&self, table: &str) -> Result<(), StoreError>;
    fn begin_read(&self) -> Result<Box<dyn StorageRoTx + '_>, StoreError>;
    fn begin_write(&self) -> Result<Box<dyn StorageRwTx + '_>, StoreError>;
    fn begin_locked(&self, table_name: &str) -> Result<Box<dyn StorageLocked>, StoreError>;
}

pub trait StorageRoTx {
    fn get(&self, table: &str, key: &[u8]) -> Result<Option<Vec<u8>>, StoreError>;
    fn prefix_iterator(
        &self,
        table: &str,
        prefix: &[u8],
    ) -> Result<Box<dyn Iterator<Item = PrefixResult> + '_>, StoreError>;
}

pub trait StorageRwTx: StorageRoTx {
    fn put(&self, table: &str, key: &[u8], value: &[u8]) -> Result<(), StoreError>;
    fn put_batch(&self, batch: Vec<(&str, Vec<u8>, Vec<u8>)>) -> Result<(), StoreError>;
    fn delete(&self, table: &str, key: &[u8]) -> Result<(), StoreError>;
    fn commit(self: Box<Self>) -> Result<(), StoreError>;
}

pub trait StorageLocked: Send + Sync + 'static {
    fn get(&self, key: &[u8]) -> Result<Option<Vec<u8>>, StoreError>;
}

TODO

Review API methods
Review all TODO and FIXME comments in the code
Re-enable tests related to TX EIP-1559
Restore previously deleted tests
Check features flags
Add documentation
Benchmark performance (ethrex_benchmarks, syncing, etc)

Closes #issue_number

…rences

…and implement it in RocksDBBackend

This reverts commit 17d0ec9.

github-actions · 2025-09-23T20:24:04Z

Lines of code report

Total lines added: 1437
Total lines removed: 327
Total lines changed: 1764

Detailed view

+-------------------------------------------------------+-------+------+
| File                                                  | Lines | Diff |
+-------------------------------------------------------+-------+------+
| ethrex/cmd/ethrex/initializers.rs                     | 387   | -2   |
+-------------------------------------------------------+-------+------+
| ethrex/cmd/ethrex/l2/command.rs                       | 653   | -7   |
+-------------------------------------------------------+-------+------+
| ethrex/cmd/ethrex_replay/src/cli.rs                   | 921   | +6   |
+-------------------------------------------------------+-------+------+
| ethrex/crates/blockchain/smoke_test.rs                | 242   | +2   |
+-------------------------------------------------------+-------+------+
| ethrex/crates/common/trie/node_hash.rs                | 109   | -15  |
+-------------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/eth/gas_price.rs         | 93    | -21  |
+-------------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/eth/gas_tip_estimator.rs | 111   | -21  |
+-------------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/eth/max_priority_fee.rs  | 63    | -36  |
+-------------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/eth/mod.rs               | 174   | -9   |
+-------------------------------------------------------+-------+------+
| ethrex/crates/storage/api.rs                          | 51    | -182 |
+-------------------------------------------------------+-------+------+
| ethrex/crates/storage/backend/in_memory.rs            | 182   | +182 |
+-------------------------------------------------------+-------+------+
| ethrex/crates/storage/backend/mod.rs                  | 3     | +3   |
+-------------------------------------------------------+-------+------+
| ethrex/crates/storage/backend/rocksdb.rs              | 237   | +237 |
+-------------------------------------------------------+-------+------+
| ethrex/crates/storage/engine.rs                       | 910   | +910 |
+-------------------------------------------------------+-------+------+
| ethrex/crates/storage/error.rs                        | 37    | -3   |
+-------------------------------------------------------+-------+------+
| ethrex/crates/storage/rlp.rs                          | 38    | -15  |
+-------------------------------------------------------+-------+------+
| ethrex/crates/storage/store.rs                        | 1508  | -16  |
+-------------------------------------------------------+-------+------+
| ethrex/crates/storage/trie.rs                         | 97    | +97  |
+-------------------------------------------------------+-------+------+

github-actions · 2025-09-24T12:25:22Z

Benchmark for `ac20fdb`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.0±0.36ms	34.8±0.24ms	-0.57%
Trie/cita-trie insert 1k	3.6±0.03ms	3.5±0.01ms	-2.78%
Trie/ethrex-trie insert 10k	55.3±1.19ms	54.7±0.34ms	-1.08%
Trie/ethrex-trie insert 1k	6.1±0.01ms	6.1±0.04ms	0.00%

github-actions · 2025-09-24T13:10:12Z

Benchmark for `515d02c`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	34.7±2.87ms	40.7±3.12ms	+17.29%
Trie/cita-trie insert 1k	3.6±0.03ms	3.6±0.02ms	0.00%
Trie/ethrex-trie insert 10k	61.4±1.55ms	57.3±1.73ms	-6.68%
Trie/ethrex-trie insert 1k	6.2±0.03ms	6.2±0.21ms	0.00%

github-actions · 2025-09-24T15:01:05Z

Benchmark for `3a6e5f0`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	37.8±2.02ms	36.5±2.26ms	-3.44%
Trie/cita-trie insert 1k	3.7±0.27ms	3.5±0.02ms	-5.41%
Trie/ethrex-trie insert 10k	60.5±1.67ms	57.6±1.48ms	-4.79%
Trie/ethrex-trie insert 1k	6.2±0.03ms	6.1±0.05ms	-1.61%

github-actions · 2025-09-24T15:17:39Z

Benchmark for `ff142bd`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.6±0.93ms	36.5±1.60ms	+2.53%
Trie/cita-trie insert 1k	3.6±0.03ms	3.7±0.20ms	+2.78%
Trie/ethrex-trie insert 10k	56.5±0.78ms	55.4±1.22ms	-1.95%
Trie/ethrex-trie insert 1k	6.1±0.09ms	6.1±0.14ms	0.00%

github-actions · 2025-09-26T15:42:36Z

Benchmark for `f30cbe6`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	39.4±1.53ms	37.6±2.02ms	-4.57%
Trie/cita-trie insert 1k	3.6±0.02ms	3.7±0.20ms	+2.78%
Trie/ethrex-trie insert 10k	57.2±1.26ms	55.5±0.90ms	-2.97%
Trie/ethrex-trie insert 1k	6.1±0.02ms	6.2±0.02ms	+1.64%

github-actions · 2025-09-26T15:44:04Z

Benchmark for `1764e7d`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	31.7±2.99ms	31.2±2.33ms	-1.58%
Trie/cita-trie insert 1k	2.7±0.01ms	2.8±0.06ms	+3.70%
Trie/ethrex-trie insert 10k	56.3±1.77ms	55.2±1.13ms	-1.95%
Trie/ethrex-trie insert 1k	5.7±0.08ms	5.7±0.03ms	0.00%

github-actions · 2025-09-26T16:08:32Z

Benchmark for `877b67f`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	34.6±2.54ms	36.2±1.97ms	+4.62%
Trie/cita-trie insert 1k	3.6±0.03ms	3.7±0.31ms	+2.78%
Trie/ethrex-trie insert 10k	57.1±1.72ms	55.9±1.74ms	-2.10%
Trie/ethrex-trie insert 1k	6.2±0.19ms	6.3±0.35ms	+1.61%

github-actions · 2025-09-30T13:40:24Z

Benchmark for `a1e0f04`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.9±0.63ms	35.7±0.69ms	-0.56%
Trie/cita-trie insert 1k	3.6±0.01ms	3.7±0.16ms	+2.78%
Trie/ethrex-trie insert 10k	55.7±0.50ms	55.5±0.67ms	-0.36%
Trie/ethrex-trie insert 1k	6.3±0.02ms	6.2±0.05ms	-1.59%

github-actions · 2025-09-30T13:52:07Z

Benchmark for `3181d80`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	36.8±1.44ms	36.3±1.17ms	-1.36%
Trie/cita-trie insert 1k	3.5±0.02ms	3.6±0.02ms	+2.86%
Trie/ethrex-trie insert 10k	55.6±0.57ms	55.7±0.59ms	+0.18%
Trie/ethrex-trie insert 1k	6.1±0.01ms	6.2±0.32ms	+1.64%

…b structs

github-actions · 2025-09-30T15:04:17Z

Benchmark for `2d4ad73`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	30.1±1.88ms	30.3±1.63ms	+0.66%
Trie/cita-trie insert 1k	2.8±0.02ms	2.8±0.02ms	0.00%
Trie/ethrex-trie insert 10k	60.0±1.50ms	62.4±3.38ms	+4.00%
Trie/ethrex-trie insert 1k	5.8±0.02ms	5.8±0.19ms	0.00%

github-actions · 2025-09-30T15:50:04Z

Benchmark for `d5ab0b2`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	38.0±1.92ms	32.7±1.26ms	-13.95%
Trie/cita-trie insert 1k	3.6±0.03ms	3.7±0.03ms	+2.78%
Trie/ethrex-trie insert 10k	57.8±0.92ms	58.1±0.84ms	+0.52%
Trie/ethrex-trie insert 1k	6.2±0.02ms	6.3±0.23ms	+1.61%

damiramirez added 30 commits September 18, 2025 16:30

feat(api): create a new api for StorageBackend

32cc7a2

feat(rocksdb): implement new interface for rocksdb

b36854c

feat(trie): add new trie db implementation

409a9a8

refactor(trie,store): use new triedb implementation in store

af9ea97

feat(store): add prefix_iterator to StorageBackend interface

ee73b03

feat(store): add clear_table to StorageBackend interface

7d7a32e

feat(storage_v2): finished with first iteration of new storage

beca284

refactor(store): use new store

f7b229c

chore(store,api): add TODOs

90a3fe3

feat(in_memory): basic implementation of in memory backend

364fbee

refactor(store): change put and delete methods to take immutable refe…

c9c9a00

…rences

feat(storem api, rocksdb): add begin_locked method to StorageBackend …

1c4e3c2

…and implement it in RocksDBBackend

fix(api): remove lifetimes in begin_locked

b1f959a

refactor(core): remove unused imports

5b098a8

refactor(store): use new version of store

6cafeaf

refactor(build_block): remove clippy warnings

70a8823

refactor(api): remove some lifetimes

20bb988

refactor(rocksdb,in_memory): use new interface

5e15cf2

refactor(trie): use new interface

515b0af

refactor(rocksdb): use a cache for CFs

d4d951e

wip

d9725d2

refactor(store): use locked trie

e31e569

fix(clippy): remove clippy warnings

ef44e44

docs(api): add documentation to StorageBackend interface

c98ca10

feat(api): add write batch to api

c80f7e1

Merge branch 'main' into feat/storage-backend-api

034fc57

feat(rocksdb): add basic DB config to check improvements

7e3bf97

refactor(api): move put batch to StorageRwTx

5880f95

feat(engine): add methods with spawn blocking

17d0ec9

Revert "feat(engine): add methods with spawn blocking"

6d9ba0b

This reverts commit 17d0ec9.

damiramirez added 4 commits September 23, 2025 16:19

fix(trie): update put batch

8b330fb

test(smoke_test): compute hash for retrieved block headers

aef76d7

test(TX-1559): comment failed tests

80b5390

Merge branch 'main' into feat/storage-backend-api

c696223

damiramirez added 3 commits September 23, 2025 17:39

refactor(replay): use new in memory storage

096f680

test(store): add store test again

b97897b

chore(libmdbx): remove libmdbx crate

d717d0f

chore(prover): build cargo lock for risc0 and sp1

5af13a7

docs(rocksdb): add documentation to the structs and specific methods

5bfec65

refactor(core): reduce diff noise

70f6f85

damiramirez added 2 commits September 26, 2025 12:35

Merge branch 'main' into feat/storage-backend-api

09a274d

feat(rocksdb): add again rocksdb options

4cf4698

refactor(rocksdb): improve prefix iterator

d788c13

refactor(rocksdb): back again to individual txs

a2714fb

fix(trie): commit after put

c5f4d19

refactor(rocksdb): use writeBatchWithTransaction as a field in rocksd…

f9a9c9e

…b structs

damiramirez added 2 commits September 30, 2025 12:17

refactor(trie): remove put from BackendTrieDB

6fbbc70

refactor(api): mut StorageRwTx methods

f2193c1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[POC] new database interface #4623

[POC] new database interface #4623

damiramirez commented Sep 23, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 23, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

github-actions bot commented Sep 26, 2025

Uh oh!

github-actions bot commented Sep 26, 2025

Uh oh!

github-actions bot commented Sep 26, 2025

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

Uh oh!

[POC] new database interface #4623

Are you sure you want to change the base?

[POC] new database interface #4623

Conversation

damiramirez commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

github-actions bot commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Lines of code report

Uh oh!

github-actions bot commented Sep 24, 2025

Benchmark for ac20fdb

Uh oh!

github-actions bot commented Sep 24, 2025

Benchmark for 515d02c

Uh oh!

github-actions bot commented Sep 24, 2025

Benchmark for 3a6e5f0

Uh oh!

github-actions bot commented Sep 24, 2025

Benchmark for ff142bd

Uh oh!

github-actions bot commented Sep 26, 2025

Benchmark for f30cbe6

Uh oh!

github-actions bot commented Sep 26, 2025

Benchmark for 1764e7d

Uh oh!

github-actions bot commented Sep 26, 2025

Benchmark for 877b67f

Uh oh!

github-actions bot commented Sep 30, 2025

Benchmark for a1e0f04

Uh oh!

github-actions bot commented Sep 30, 2025

Benchmark for 3181d80

Uh oh!

github-actions bot commented Sep 30, 2025

Benchmark for 2d4ad73

Uh oh!

github-actions bot commented Sep 30, 2025

Benchmark for d5ab0b2

Uh oh!

Uh oh!

damiramirez commented Sep 23, 2025 •

edited

Loading

github-actions bot commented Sep 23, 2025 •

edited

Loading

Benchmark for `ac20fdb`

Benchmark for `515d02c`

Benchmark for `3a6e5f0`

Benchmark for `ff142bd`

Benchmark for `f30cbe6`

Benchmark for `1764e7d`

Benchmark for `877b67f`

Benchmark for `a1e0f04`

Benchmark for `3181d80`

Benchmark for `2d4ad73`

Benchmark for `d5ab0b2`