CAIN-42 evidence library
Cluster restore — 2026-11-16
Evidence bundle: 0 claims, 0 of 0 invariants held.
Last reviewed 2026-11-19
The bundle's own README
Cluster restore — 2026-11-16#
Two PBFT clusters were degraded. Both are back to 4 of 4 healthy replicas with quorum-valid certificates. This bundle is the evidence of what was wrong, what was changed, and what is still not proven.
- Live cluster:
cain-mr-03(N=4, f=1, Q=3), four regions, one replica each. - Second cluster:
cain-mr-01(N=4, f=1, Q=3), also monitored and served by the gateway. - Retired cluster:
cain-mr-02, quarantined 2026-11-15 for equivocation
(docs/audits/PBFT-SAFETY-INCIDENT-2026-11-15.md). It stays stopped.
Nothing here is a third-party audit or a certification. The 48-hour soak is at 0.01 of 48 h, not passed. Evidence class: self-observed, same-admin-domain.
1. What was found#
1.1 cain-mr-03 had lost a replica (root cause: the guardian revived a retired node)#
The Moat #7 Sentinel (cain-sentinel-guardian.service) still listed the retired, quarantined container cain-mr2-node-1 as a required container. At 2026-11-16T10:54:29Z it found that container stopped and auto-started it.
cain-mr-02 and cain-mr-03 were built on the same overlay addresses and ports (10.88.0.1:29811 … 10.88.0.4:29814). The revived cain-mr2-node-1 took 10.88.0.1:29811 from the live replica, and at 11:04:17Z cain-mr3-node-1 exited with code 3:
ERROR: [Errno 98] error while attempting to bind on address ('10.88.0.1', 29811): address already in use
The guardian had degraded the cluster it was guarding. The live cluster ran on 3 of 4 replicas (quorum was still met, so it did not halt) while the 48-hour soak kept logging CHECK ERROR (state comparison skipped).
The retired container was revived once more by the old guardian at 11:35:26Z, seconds after it was stopped and about a minute before the guardian was restarted with the fix. It could not bind and died at 11:48:06Z. It has not been started since; the retired-container guard is exercised directly in post-fix/retired-guard-proof.txt.
1.2 cain-mr-01 was stalled and committing without quorum#
A separate, more serious fault. Replica heights were frozen at different values:
| Replica | commit_index | health |
| node-1 | 143271 | HEALTHY |
| node-2 | 143230 | HEALTHY |
| node-3 | 143230 | HEALTHY |
| node-4 | 143035 | SAFE_MODE |
node-4 carried wal_write_failure: "sequence 143035: OperationalError: disk I/O error" and had been in SAFE_MODE since that write, with no restart since 2026-11-03.
A write sent to node-1 returned:
{"status": "COMMITTED", "decision": "ALLOW",
"quorum_certificate_status": "INVALID: INSUFFICIENT_QUORUM",
"quorum_certificate_accepted_signers": 1, "sequence": 143273}
So the node reported COMMITTED while only one replica had signed, below the quorum of 3. The local commit is real (the height advanced); the outcome was not quorum-certified. The response made that visible only in an adjacent string field — a caller reading status alone would treat an uncertified outcome as final. That is the fail-open reporting defect recorded in section 3.2.
After the repair the same write returns "quorum_certificate_status": "VALID" with 4 accepted signers (mr01/post-fix-write-probes.txt).
2. What was done#
| # | Change | File |
| 1 | Stopped the invalid soak run | systemctl stop cain-mr03-soak-48h-2026-11-15b |
| 2 | Stopped the revived retired container (container and data preserved) | docker stop cain-mr2-node-1 |
| 3 | Required container cain-mr2-node-1 → cain-mr3-node-1 | deploy/scripts/cain_sentinel_guardian.py |
| 4 | The guardian now refuses to start any retired cain-mr2-node-* container | same |
| 5 | Cluster monitor cain-mr-02 → cain-mr-03 | same |
| 6 | Gateway knows cain-mr-03 | platform-gateway/routers/live_cluster.py |
| 7 | Gateway refuses to count a replica whose reported node_id/cluster_id does not match the requested cluster | same |
| 8 | outcome_certified / outcome_final added to the commit response (staged; not live until the node image is rebuilt) | platform-gateway/cain_pbft_engine_33.py |
| 9 | Restarted cain-mr3-node-1; it caught up from its peers | docker start cain-mr3-node-1 |
| 10 | Restarted the SAFE_MODE replica cain-mr-01/node-4; it caught up 143035 → 143275 | docker restart cain-mr-node-4 |
| 11 | Started a new, clean 48-hour soak with a new name | cain-mr03-soak-48h-2026-11-16c |
Change 7 matters for honesty: before it, GET /api/v1/live-cluster/health?cluster=cain-mr-02 reported 3 of 4 replicas reachable at height 11468 for a cluster that no longer exists — it was counting cain-mr-03's replicas, which answer on the same overlay addresses. After the change the retired cluster reports HALTED, 0/4 and the live cluster reports HEALTHY, 4/4.
3. Verification after the repair#
3.1 Live cluster cain-mr-03#
All four replicas: view 59→, identical commit_index, identical application_state_hash, health_state: HEALTHY. A write returns VALID with 4 accepted signers.
3.2 Second cluster cain-mr-01#
All four replicas at commit 143275 and advancing; three consecutive writes return VALID with 4 accepted signers. Host free disk at repair time: atl 241 GB, lax 9.2 GB, mia 26 GB, sjc 62 GB.
3.3 Soak#
New run soak-multiregion-48h-2026-11-16-mr03c started 2026-11-16T11:48:09Z. Its preflight passed (all four containers running, all HEALTHY at one height and state, 12-sequence decision window readable from at least two replicas with no disagreement). checkpoint-0000.json is VALID, 48 certificates verified; verdict RUNNING (0.009 of 48.0 h).
The previous run soak-multiregion-48h-2026-11-15-mr03b is marked INVALID in its own bundle (INVALID-RUN.md): it is 18 h of a degraded window, not a 48-hour run, and must not be cited.
4. Follow-ups (fixed the same day)#
Two open items from this restore were fixed on 2026-11-16 and are recorded in followups/:
1. The soak verdict is no longer divergences == 0 alone. It now fails on a sustained streak of unreadable checks with no injected fault, on a cluster that is not healthy at one height when the run ends, and on a host below the new --min-free-gb-run floor. verdict() is a pure function; tests/test_soak_verdict.py is 8/8. 2. The lax host was reclaimed from 9.23 GB to 14.94 GB by removing four clean, reproducible verification clones. The quarantined cain-mr-02 data (20.5 GB) was not touched.
The run that enforces the new verdict is soak-multiregion-48h-2026-11-16-mr03d (started 12:05:15Z, checkpoint-0000 VALID). It is 0.01 of 48 h.
5. What this does not prove#
- Not a third-party audit, not a certification, not a SOC 2 or ISO claim.
- The 48-hour soak has not passed. It is 0.01 h in and will restart from zero if it is
interrupted. Do not publish a 48h or 72h claim from this bundle.
- The
wal_write_failureoncain-mr-01/node-4was cleared by a restart. **The underlying I/O
error was not root-caused.** Miami has 26 GB free now; the error is consistent with the host having been full earlier (the same host filled up once before), but that is not established here.
- The
outcome_certified/outcome_finalfields are in the host source only. The running node
image (cain-cluster-node:mr-214970f) still has the old response shape until it is rebuilt and redeployed. The invalid-quorum state is documented, not prevented.
cain-mr-01andcain-mr-03are both operated from this same administrative domain. They are
not independent witnesses of each other.
cain-mr-02remains quarantined. Its evidence is untouched; it was not repaired and must not be
used as a trust anchor.
6. Reproduce this check#
python3 verify_restore_2026_11_16.py # verifies MANIFEST.json and the recorded facts curl -s 'http://127.0.0.1:8088/api/v1/live-cluster/health?cluster=cain-mr-03' curl -s 'http://127.0.0.1:8088/api/v1/live-cluster/health?cluster=cain-mr-01' systemctl is-active cain-mr03-soak-48h-2026-11-16c.service python3 /root/cain-studio-platform/scripts/cain42_bft/verify_soak.py \ /root/cain-studio-platform/platform-gateway/frontend/proof/bundle/soak-multiregion-48h-2026-11-16-mr03c
7. Files#
| File | Contents |
RESTORE.json | machine-readable summary: findings, changes, before/after states |
pre-fix/ | docker state, exit log, sentinel auto-heal log, soak tail, per-replica status before |
mr01/ | cain-mr-01 before-fix status, the invalid-quorum response, after-fix write probes |
followups/ | soak-verdict fix and tests, disk reclaim before/after, the new run's checkpoint-0000 |
post-fix/ | per-replica status, gateway health for both clusters, sentinel state, soak verification |
RECOVERY-COMMANDS.txt | the exact commands run, in order |
verify_restore_2026_11_16.py | standalone verifier, imports no CAIN code |
MANIFEST.json | SHA-256 of every file in this bundle |
Try CAIN-42 on your own agents
Create a free account and every new account starts with a 30-day trial of the full platform. Or try the sandbox first, with no account at all.
Create a free account → · Try the sandbox · See the whole ecosystem