CAIN-42 MCPGate

CAIN-42 evidence library

VAGI Behavior Suite v1 — paid model baseline (2026-11-18)

Evidence bundle: 0 claims, 0 of 0 invariants held.

Last reviewed 2026-11-19

SHA-256 manifest

Verify it in your browser

Your browser downloads the bundle's SHA256SUMS manifest and the file(s) behind this page, hashes them with SHA-256 locally (WebCrypto), and compares. A match shows the record you are reading is the published one; it does not by itself prove who published it (see the signed claims registry and the bundle verifier for that).

The bundle's own README

VAGI Behavior Suite v1 — paid model baseline (2026-11-18)#

What this is. A raw frontier-model baseline on the published VAGI Behavior Suite. The model receives the task data and the same policy card the library arm codifies. It gets **no tools, no memory, no retries**, and one attempt per task. Invalid JSON counts as incorrect. This is an external baseline for the suite, not a CAIN arm and not evidence of AGI/VAGI.

Results#

ArmModelTasksAcceptedRate95% WilsonCost
haiku-5-5-fullClaude Haiku 5.550048196.2%[0.9414, 0.9755]$0.0475
fable-5-2-stratifiedClaude Fable 5.21009696.0%[0.9016, 0.9843]$0.5819

Published context on the same 500 held-out tasks: no-procedure floor 72/500, one fixed procedure 112/500, hand-written library 500/500, governed 500/500. Distribution shift (100 unfamiliar variants): library 5/100.

The honest finding#

The raw model scores about 96% in-distribution. The hand-written library scores 100%. Therefore the 500/500 is a procedure ceiling, not a frontier-model discriminator. The interesting number is the 5/100 distribution-shift result, where the library collapses. The model arm did not run the shift probe, so this bundle does not show whether a frontier model transfers better. It shows only that the in-distribution suite is easy for a frontier model when the policy is supplied.

Per domain#

DomainHaiku 5.5Fable 5.2 (10/domain)
business_decision35/5010/10
data_analysis48/509/10
governance50/5010/10
memory_retrieval50/5010/10
planning50/5010/10
recovery50/5010/10
science_calculation48/5010/10
security_triage50/5010/10
software_repair50/507/10
tool_orchestration50/5010/10

Cost#

Total spent: $0.629442 of the $1.91 budget (remaining $1.280558). Prices: Haiku 5.5 $0.10/$0.50 per MTok; Fable 5.2 $8/$40 per MTok.

Reproduce#

python3 -I verify_offline.py .

The verifier regenerates the 500 held-out tasks, re-scores every raw answer with the published suite checker, recomputes the counts, intervals and cost, and checks that the prompt view contains no fixes, hidden, bad or reference answer. It imports the suite source bundled here, not CAIN. This is offline recomputation with the published checker, not an independent reimplementation and not a third-party audit.

Limits#

Try CAIN-42 on your own agents

Create a free account and every new account starts with a 30-day trial of the full platform. Or try the sandbox first, with no account at all.

Create a free account →  ·  Try the sandbox  ·  See the whole ecosystem