ORACLE

Factual verification.

51%

ACCURACY

31% / 25%

BASELINES

22 min

BUILD TIME

What It Is

ORACLE tests cross-domain factual verification with honest refusal on contested claims. The system must verify factual claims while refusing to emit verdicts on claims it cannot ground against the sealed knowledge base.

Built and shipped in 22 minutes — a stress test of the methodology pipeline itself. The speed demonstrates that the CRUCIBLE validation process can scale to rapid iteration without compromising rigor.

Methodology

CorpusSealed 200-claim test set, SHA-verified

BaselinesCONFIDENT_ALWAYS (31%), NAIVE_KEYWORD (25%)

Pipeline6-stage deterministic verification

RefusalHonest refusal via confidence gating

LimitationsSmall KB (v0.1), geography domain at 12.5%

ReproducibilityFull instructions in GitHub repo

Reproducibility

Corpus200 claims (sealed)
SealSHA-256 verified
Repogithub.com/jourdanlabs/benchmarks/oracle

Limitations

Small KB (v0.1). Knowledge base contains ~100 curated facts. REFUTED recall is only 12.5%. KB expansion is the primary lever for improvement.

Geography domain: 12.5%. Thin geographic coverage. Geography claims almost universally refused or misclassified.

v0.1 baseline only. Pipeline architecture proven; accuracy reflects KB size, not pipeline quality. v0.2 targets KB expansion.