Database Operation Transition: From Badger to Pebble

[zhangchiqing]

Transition the database operation implementation from Badger-based to Pebble-based.

This task is dependent on the refactor that changes from Badger transactions to Badger batch updates.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[j1010001]

Discussed approach in execution team meeting Mar 10: https://www.notion.so/flowfoundation/Execution-Team-meeting-agenda-82cc3be512df4497a1a7ce57cc786a40?pvs=4#1af1aee123248001ac26e5f553f21762

[zhangchiqing]

For access nodes, once switching storage from badger to pebble, new data will be stored to pebble database, such as transaction, collections, events, transaction results etc. The switch would also make the data query to look up from pebble database. However, this would cause issue if the query is to find the old data previously stored in badger, and return not found by mistake.

To resolve it, we need to build a layer on top of the storage layer, so that once switching to pebble, it can query both badger to pebble for retrieving data. Note, this layer is only needed after the switch, it's not needed before the switch, since all the data are still stored badger.

[fxamacker]

Issue text says:

Transition the database operation implementation from Badger-based to Pebble-based.

I will take a look at the operation layer code and put together a list of any remaining work.

[fxamacker]

Updates #6527 (milestone 2)

This issue was closed with the understanding its scope is limited to #6523 (comment) and these PRs completing that aspect.

• [Storage] Add database multiReader, multiIterator, multiSeeker (BadgerDB, Pebble) #7320
• [Access] Add support for multi-store reading of collections & transactions in Access Nodes #7321
• [Access] Add support for multi-store reading of events, lightTransactionResults, etc. (BadgerDB & Pebble) #7335

When time allows, maybe Leo or I can take a look at the operation layer implementation to see if more work remains in the broader scope of "Transition the database operation implementation from Badger-based to Pebble-based". We can open a separate issue with subtasks if we find any remaining work that was missed.