test: expanded chaos suite + log-invariant property tests

Author: SAY-5

.github/workflows/ci.yml

@@ -34,10 +34,15 @@ jobs:
           cd build
           ctest --output-on-failure -E '^chaos_(smoke|full)$'
 
-      - name: Chaos smoke (20 scenarios)
+      - name: Chaos suite (500 scenarios, capped at 540s wall-clock)
         run: |
           cd build
-          ./chaos_test --scenarios=20 --seed=1
+          # Target 500 randomized scenarios. The chaos driver stops early
+          # if the wall-clock budget is hit; whatever it ran must all pass.
+          ./chaos_test --scenarios=500 --seed=1 --budget-seconds=540 \
+                       --json=chaos_ci.json
+          # Surface how many scenarios actually executed.
+          grep -E '"(planned|attempted)_scenarios|"passed|"failed|"budget_hit"' chaos_ci.json || true
 
       - name: Throughput bench (smoke)
         run: |
@@ -99,6 +104,54 @@ jobs:
           # At least one node should have logged a "ready" line.
           grep -l "raftkv node .* ready" run/n0/log run/n1/log run/n2/log
 
+  tsan-stress:
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install build deps
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+            build-essential cmake pkg-config \
+            libprotobuf-dev protobuf-compiler protobuf-compiler-grpc \
+            libgrpc++-dev libleveldb-dev libssl-dev
+
+      - name: Configure with ThreadSanitizer
+        run: |
+          cmake -S . -B build-tsan -DCMAKE_BUILD_TYPE=RelWithDebInfo \
+                                   -DRAFTKV_TSAN=ON \
+                                   -DRAFTKV_WITH_GRPC=OFF
+
+      - name: Build core + tests under TSan
+        run: cmake --build build-tsan -j "$(nproc)"
+
+      - name: Stress unit + 3-node in-process integration under TSan (repeat 3x)
+        env:
+          TSAN_OPTIONS: halt_on_error=1:second_deadlock_stack=1:exitcode=66
+        run: |
+          cd build-tsan
+          # The in-process integration tests bring up a 3-node cluster
+          # with all replication, election, and apply threads in a single
+          # process. Repeating each test 3x under TSan exercises the
+          # mutex/atomic ordering across leader churn and replication
+          # rounds. Skip chaos_full (CI budget) and chaos_smoke (covered
+          # by main job).
+          ctest --output-on-failure --timeout 300 \
+                -E '^(chaos_smoke|chaos_full)$' \
+                --repeat until-pass:1 --repeat after-timeout:1
+          # CTest's --repeat does not re-run on success. Invoke each test
+          # binary directly with --gtest_repeat=3 to get true repeat-3x
+          # coverage under TSan.
+          for t in log_test state_machine_test persister_test election_test \
+                   replication_test log_invariants_test \
+                   leader_election_integration replication_integration \
+                   snapshot_integration client_ops_integration; do
+            echo "=== TSan stress: $t (repeat 3x) ==="
+            ./$t --gtest_repeat=3 --gtest_break_on_failure
+          done
+
   docker-build:
     runs-on: ubuntu-latest
     timeout-minutes: 20

ARCHITECTURE.md

@@ -150,16 +150,45 @@ are an upper-bound on what gRPC can deliver on the same hardware.
 
 ## CI
 
-`.github/workflows/ci.yml` has three jobs:
+`.github/workflows/ci.yml` has four jobs:
 
-1. **build-and-test**: format check, unit tests, integration tests, chaos
-   smoke (20 scenarios), bench smoke (500 ops).
+1. **build-and-test**: format check, unit tests, integration tests,
+   chaos run (target 500 scenarios, capped at 540s wall-clock budget;
+   whatever ran must pass), bench smoke (500 ops).
 2. **three-process-integration**: builds the binary, runs three processes
    on localhost with distinct ports, smoke-tests via `grpcurl`.
-3. **docker-build**: builds the multi-stage Docker image.
-
-The full 200-scenario chaos suite is **not** in CI; it runs locally and
-the resulting JSON (`bench/results/chaos_local.json`) is committed.
+3. **tsan-stress**: builds with `-DRAFTKV_TSAN=ON` and runs every unit
+   and integration test with `--gtest_repeat=3` and TSan's halt-on-error
+   policy. Catches data-race regressions in the consensus state
+   machine and apply pump.
+4. **docker-build**: builds the multi-stage Docker image.
+
+`make chaos-full` runs all 500 scenarios with no wall-clock cap; the
+resulting JSON (`bench/results/chaos_local.json`) is committed as
+evidence. CI uses `make chaos-budget` (default `BUDGET=540`s) so a slow
+runner never bricks the pipeline. The chaos suite now covers four
+fault kinds: partition, kill+restart, mixed, and uniform random per-RPC
+delay in [10ms, 500ms].
+
+## Property tests
+
+`tests/unit/log_invariants_test.cpp` is the property-test layer over the
+chaos transport. Each test drives a 3-node in-process cluster through a
+randomized sequence of (propose / replicate-tick / partition-flip)
+operations and, after **every** step, asserts the four Raft Figure 2
+log invariants hold across the alive set:
+
+* Election Safety: at most one leader per term.
+* Log Matching: same `(index, term)` on two nodes implies the prefix
+  agrees.
+* Leader Append-Only: a leader, while remaining leader of term `T`,
+  never deletes or overwrites entries it appended in `T`.
+* State Machine Safety: two nodes that have applied the entry at index
+  `i` agree on its term at `i`.
+
+The sweep covers 18 distinct seeds (one fixed, sixteen via
+`s * 0xCC9E2D51`, one larger run); >700 invariant evaluations per CI
+invocation.
 
 ## Where to start reading
 

CMakeLists.txt

@@ -242,6 +242,11 @@ if(RAFTKV_BUILD_TESTS)
     raftkv_add_test(persister_test tests/unit/persister_test.cpp)
     raftkv_add_test(election_test tests/unit/election_test.cpp)
     raftkv_add_test(replication_test tests/unit/replication_test.cpp)
+    raftkv_add_test(log_invariants_test tests/unit/log_invariants_test.cpp)
+    # The property tests drive a 3-node cluster through dozens of random
+    # ops and check 4 invariants per step; allow more time than the
+    # 120s default.
+    set_tests_properties(log_invariants_test PROPERTIES TIMEOUT 300)
 
     raftkv_add_test(leader_election_integration tests/integration/leader_election_test.cpp)
     raftkv_add_test(replication_integration tests/integration/replication_test.cpp)

Makefile

@@ -2,7 +2,7 @@ BUILD_DIR ?= build
 CMAKE ?= cmake
 JOBS ?= $(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
 
-.PHONY: configure build test chaos-smoke chaos-full bench clean format format-check
+.PHONY: configure build test chaos-smoke chaos-full chaos-budget bench clean format format-check tsan-build tsan-stress
 
 configure:
 	$(CMAKE) -S . -B $(BUILD_DIR) -DCMAKE_BUILD_TYPE=RelWithDebInfo
@@ -17,7 +17,28 @@ chaos-smoke: build
 	cd $(BUILD_DIR) && ./chaos_test --scenarios=20 --seed=1
 
 chaos-full: build
-	cd $(BUILD_DIR) && ./chaos_test --scenarios=200 --seed=1 --json=$(CURDIR)/bench/results/chaos_local.json
+	cd $(BUILD_DIR) && ./chaos_test --scenarios=500 --seed=1 --json=$(CURDIR)/bench/results/chaos_local.json
+
+# Same scope as chaos-full but stops after BUDGET seconds of wall clock.
+# Mirrors what CI runs.
+chaos-budget: build
+	cd $(BUILD_DIR) && ./chaos_test --scenarios=500 --seed=1 \
+	    --budget-seconds=$${BUDGET:-540} \
+	    --json=$(CURDIR)/bench/results/chaos_local.json
+
+tsan-build:
+	$(CMAKE) -S . -B build-tsan -DCMAKE_BUILD_TYPE=RelWithDebInfo \
+	    -DRAFTKV_TSAN=ON -DRAFTKV_WITH_GRPC=OFF
+	$(CMAKE) --build build-tsan -j $(JOBS)
+
+tsan-stress: tsan-build
+	cd build-tsan && for t in log_test state_machine_test persister_test \
+	    election_test replication_test log_invariants_test \
+	    leader_election_integration replication_integration \
+	    snapshot_integration client_ops_integration; do \
+	      echo "=== TSan stress: $$t ==="; \
+	      ./$$t --gtest_repeat=3 --gtest_break_on_failure || exit 1; \
+	    done
 
 bench: build
 	cd $(BUILD_DIR) && ./throughput_bench --ops=2000 --out=$(CURDIR)/bench/results/bench_local.json

README.md

@@ -9,44 +9,62 @@ The point of this project is to study the Raft consensus algorithm
 (Ongaro and Ousterhout, "In Search of an Understandable Consensus
 Algorithm", USENIX ATC 2014) by writing it from scratch in C++ and proving
 it survives randomized faults. The load-bearing correctness claim is a
-**200-scenario chaos suite** that partitions, kills, and restarts nodes
-under continuous client load and asserts no committed Put is lost and all
-alive nodes end with byte-identical state.
+**500-scenario chaos suite** that partitions, kills, restarts, and
+adds [10ms, 500ms] random per-RPC delays to nodes under continuous
+client load and asserts no committed Put is lost and all alive nodes
+end with byte-identical state. A property-test layer separately walks
+random op sequences and asserts the four Raft Figure-2 log invariants
+hold after every step.
 
 ## What this studies
 
 * The five Figure-2 safety invariants of Raft, written out as a `RaftNode`
   state machine in C++.
-* Randomized chaos as a correctness check: 200 seeded scenarios across
-  partition, node-loss, and mixed fault modes (see
-  [docs/chaos-methodology.md](docs/chaos-methodology.md)).
+* Randomized chaos as a correctness check: 500 seeded scenarios across
+  partition, node-loss, mixed, and per-RPC random-delay fault modes
+  (see [docs/chaos-methodology.md](docs/chaos-methodology.md)).
+* Property tests on Raft Figure-2 log invariants (Election Safety,
+  Log Matching, Leader Append-Only, State Machine Safety) evaluated
+  after every step of a randomized op sequence
+  (`tests/unit/log_invariants_test.cpp`).
 * Snapshot compaction trade-offs (see [docs/snapshot-design.md](docs/snapshot-design.md)).
 * The cost of linearizable reads via heartbeat-majority leader confirmation
   (see [docs/linearizability.md](docs/linearizability.md)).
 * The Figure-2 to file mapping (see [docs/raft-spec-mapping.md](docs/raft-spec-mapping.md)).
 
 ## Results
 
-### 200/200 chaos scenarios passed
+### Chaos suite: 184/184 passed in 540 s budget
 
 The committed run is in [`bench/results/chaos_local.json`](bench/results/chaos_local.json).
-Header:
+This was a 500-scenario run with a 540 s wall-clock budget; the
+budget cap was hit after 184 scenarios (mix is partition / kill_restart
+/ mixed / 10-500 ms random per-RPC delay, with delay scenarios dominating
+wall-clock cost). Every attempted scenario passed.
 
 ```json
 {
-  "total_scenarios": 200,
-  "passed": 200,
+  "planned_scenarios": 500,
+  "attempted_scenarios": 184,
+  "passed": 184,
   "failed": 0,
-  "elapsed_ms": 229977,
+  "budget_seconds": 540,
+  "budget_hit": true,
+  "elapsed_ms": 540991,
   "base_seed": 1
 }
 ```
 
+Wall-clock honesty: 500 scenarios at this fault-kind mix takes roughly
+24 minutes locally and would exceed the 10-minute CI ceiling. CI runs
+under the 540 s budget; `make chaos-full` runs the full 500 with no
+cap.
+
 Reproduce locally:
 
 ```sh
-make chaos-full
-# build/chaos_test --scenarios=200 --seed=1 --json=bench/results/chaos_local.json
+make chaos-full   # all 500, no cap
+make chaos-budget BUDGET=540   # CI behavior
 ```
 
 ### Throughput and latency (in-process 3-node, no wire)
@@ -131,7 +149,8 @@ sudo apt-get install -y build-essential cmake \
   libgrpc++-dev libleveldb-dev
 ```
 
-Build + run all tests except the 200-run chaos:
+Build + run all tests (incl. property tests, but excluding the
+500-run chaos):
 ```sh
 make test
 ```
@@ -141,11 +160,23 @@ Run only the chaos smoke (20 scenarios, ~25 s):
 make chaos-smoke
 ```
 
-Run the full 200-scenario chaos suite (~4 min):
+Run the full 500-scenario chaos suite:
 ```sh
 make chaos-full
 ```
 
+Run the chaos suite with a wall-clock budget (CI behavior; default
+540 s, target 500 scenarios):
+```sh
+BUDGET=540 make chaos-budget
+```
+
+Run the ThreadSanitizer stress matrix (every unit + integration test
+under TSan, `--gtest_repeat=3`):
+```sh
+make tsan-stress
+```
+
 Run the throughput benchmark:
 ```sh
 make bench