Incorrect transaction receipts after reorg on Holesky

[quickchase]

System information

Geth version: Geth/v1.13.11-stable-8f7eb9cc/linux-amd64/go1.21.6
CL client & version: Lighthouse v4.6.0-1be5253
OS & Version: Linux
Commit hash : (if develop)

Expected behaviour

Return the correct transaction receipt after reorg

Actual behaviour

The node returns the incorrect receipt, from the forked block, not the real block.

This only happens SOMETIMES, not always. I have multiple geth full nodes on Holesky and they frequently disagree about transaction receipt details.

Steps to reproduce the behaviour

This is VERY hard to reproduce and you might need multiple geth nodes to query to find it. However, it's happening rather frequently. We ask for a transaction receipt of a given transaction that might be in a fork block and even though the node has already re-orged back to the correct chan, the receipt doesn't get updated...

Ok, here's what I can see:

Take this transaction hash: 0xa40c2c133a9ebdb9324b2045ca3662fcb064643733fa1d272a951ad9603b578d

https://holesky.etherscan.io/tx/0xa40c2c133a9ebdb9324b2045ca3662fcb064643733fa1d272a951ad9603b578d

It can be found in this block: https://holesky.etherscan.io/block/863089

But it can also be found in this block: https://holesky.etherscan.io/block/863089/f

On the "problem node" here's the proof:

# Get forked block by hash and grep the txhash:
curl -s -H 'Content-Type: application/json' http://localhost:8545 -d '{"jsonrpc":"2.0","id":1,"method":"eth_getBlockByHash","params":["0x35700cb7eb316e36bba0f0d77124026cb0d0e2438f37164485705e1b535d299e",false]}' | jq '.result.transactions[]' | grep 0xa40c2c133a9ebdb9324b2045ca3662fcb064643733fa1d272a951ad9603b578d
"0xa40c2c133a9ebdb9324b2045ca3662fcb064643733fa1d272a951ad9603b578d"

# Get the real block by hash and grep the txhash:
curl -s -H 'Content-Type: application/json' http://localhost:8545 -d '{"jsonrpc":"2.0","id":1,"method":"eth_getBlockByHash","params":["0x60564dc65c29b542a2d9bc17cf14a613b0694cc50e8f865caf1bba450209b883",false]}' | jq '.result.transactions[]' | grep 0xa40c2c133a9ebdb9324b2045ca3662fcb064643733fa1d272a951ad9603b578d
"0xa40c2c133a9ebdb9324b2045ca3662fcb064643733fa1d272a951ad9603b578d

So the node knows about both the fork and the real block and the txhash is in both, ask for the receipt:

curl -s -H 'Content-Type: application/json' http://localhost:8545 -d '{"jsonrpc": "2.0", "method": "eth_getTransactionReceipt", "params": ["0xa40c2c133a9ebdb9324b2045ca3662fcb064643733fa1d272a951ad9603b578d"], "id": 1}' | jq
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "blockHash": "0x35700cb7eb316e36bba0f0d77124026cb0d0e2438f37164485705e1b535d299e",
    "blockNumber": "0xd2b71"

The node is clearly returning the receipt of the fork... but if I ask the node for the block by number, we can see, it's not on the fork...

curl -s -H 'Content-Type: application/json' http://localhost:8545 -d '{"jsonrpc":"2.0","id":1,"method":"eth_getBlockByNumber","params":["0xd2b71",false]}' | jq '.result.hash'
"0x60564dc65c29b542a2d9bc17cf14a613b0694cc50e8f865caf1bba450209b883"

Thus it is my belief that there is some bug in Geth preventing the receipt data from getting updated after the fork and returning the correct response.

Again, this does not happen on all my nodes at the same time, but it does happen somewhat randomly.

Geth config

--state.scheme=path

[lightclient]

Duplicate of #28885 solved in #28865.