[BUG] Cluster crashed after deleting a key

[yzhao244]

Describe the bug

In cluster, if a slave just connects to a master, a full synchronization would occur for the Slave to sync data to its own storage. However, the slots_to_keys radix tree of the Slave is not being built at all during the process of full sync. If a failover happened and the Slave promotes to become the new Master, its slots_to_keys tree is still empty. Therefore, a delete key fails to delete its key out of an empty slots_to_keys and eventually leads to a crash.

To reproduce

In cluster mode, just right after full synchronization occurred and if the Slave got promoted to be the new master due to failover , the node is crushed when trying to delete a key.

1. Start cluster.
2. Write any key such as set {a}1 123 to the master node.
3. Add a slave node to the master node. Full sync happened
4. Run "cluster failover" on Slave to trigger a master/slave failover
5. Delete {a}1 from the new master node.
6. The new master node crashed due to deleting a key out of an empty slots_to_keys tree

5211:5410:S 01 Mar 2023 19:36:55.648 * MASTER <-> REPLICA sync started
5211:5410:S 01 Mar 2023 19:36:55.649 * Non blocking connect for SYNC fired the event.
5211:5410:S 01 Mar 2023 19:36:55.650 * Master replied to PING, replication can continue...
5211:5410:S 01 Mar 2023 19:36:55.650 * Trying a partial resynchronization (request 3afea6fed57dde9b4bd2a29b7e6627c98ed294d7:1).
5211:5410:S 01 Mar 2023 19:36:56.458 * Full resync from master: 6cbe39aaf277c7a760e965eff5e5bbef8c68527d:0
5211:5410:S 01 Mar 2023 19:36:56.458 * Discarding previously cached master state.
5211:5410:S 01 Mar 2023 19:36:56.459 * MASTER <-> REPLICA sync: Flushing old data
5211:5410:S 01 Mar 2023 19:36:56.474 * Fast sync complete
5211:5410:S 01 Mar 2023 19:36:56.475 * MASTER <-> REPLICA sync: Finished with success
5211:5410:S 01 Mar 2023 19:38:22.207   audit: receive "cluster FAILOVER" from trusted network, client: 172.25.0.99:49110(:0), db:0
5211:5410:S 01 Mar 2023 19:38:22.207 # Manual failover user request accepted.
5211:5410:S 01 Mar 2023 19:38:22.208 # Received replication offset for paused master manual failover: 135
5211:5410:S 01 Mar 2023 19:38:22.208 # All master replication stream processed, manual failover can start.
5211:5410:S 01 Mar 2023 19:38:22.208 # Start of election delayed for 0 milliseconds (rank #0, offset 135).
5211:5410:S 01 Mar 2023 19:38:22.208 # Starting a failover election for epoch 4.
5211:5410:S 01 Mar 2023 19:38:22.209 # Failover election won: I'm the new master.
5211:5410:S 01 Mar 2023 19:38:22.209 # configEpoch set to 4 after successful failover
5211:5410:S 01 Mar 2023 19:38:22.209 # Connection with master lost.
5211:5410:S 01 Mar 2023 19:38:22.209 * Caching the disconnected master state.
5211:5410:S 01 Mar 2023 19:38:22.210 * Discarding previously cached master state.
5211:5410:S 01 Mar 2023 19:38:22.210 # Setting secondary replication ID to 6cbe39aaf277c7a760e965eff5e5bbef8c68527d, valid up to offset: 136. New replication ID is bcb728ad9e81d715530096010fd6a905f3ee4b10
5211:5410:M 01 Mar 2023 19:38:22.225   audit: receive "psync 6cbe39aaf277c7a760e965eff5e5bbef8c68527d 136" from trusted network,
5211:5410:M 01 Mar 2023 19:38:22.225 * Replica 172.25.0.99:8020 asks for synchronization
5211:5410:M 01 Mar 2023 19:38:22.225 * Partial resynchronization request from 172.25.0.99:8020 accepted. Sending 0 bytes of backlog starting from offset 136.

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
5211:5410:M 01 Mar 2023 19:38:28.436 # === ASSERTION FAILED ===
5211:5410:M 01 Mar 2023 19:38:28.436 # ==> db.cpp:2537 'fModified || g_pserver->db[0]->snapshot_depth() > 0' is not true

Expected behavior

In some way, slots_to_keys in cache may have to sync up with keys only available in storage layer after full sync.

1. Build slots_to_keys during full sync
2. Asynchronously build slots_to_keys in some way after the full sync.

Additional information

Any additional information that is relevant to the problem.

[paulmchen]

@JohnSully @msotheeswaran cc John and Malavan