IVChoicesDefaults, departures, and the rows that record them: a protocol for testing claims of machine choice against a production record, five choices that survive it, and the test it cannot yet run. · Read the paper: Choices ↗
The Substrate series · Rajaii Labs
I
Paper I,
Recall, Negation, Abstention: Evaluating a Production LLM Agent Against Its Own Failure Modes
A field methodology for hallucination, contradiction, and drift detection in memory-augmented agents.
The evaluation method built while running Substrate in continuous production: a seven-class failure taxonomy drawn from real incidents, a six-layer honesty architecture that moved from recall to lookup, a cold-run probe suite for recall, negation and abstention, and an incident forensic showing why ‘hallucination’ is a symptom label rather than a diagnosis.
Granting autonomy to a production LLM agent, the specification it wrote for its own interior, and five silent defects found by following the money.
The agent wrote a build specification for read access to its interior, with scope, acceptance criteria and a negative test, and the human role on that build shrank to transport, review and the gate. Following the money surfaced five defects that never raised an error, and an optimization that passed its tests while making every warm turn 21.6% more expensive.
Persistence, forgetting, self-models, dreams: the buildable half of the question.
Machine self-awareness is usually argued as one untestable threshold. This paper breaks it into architectural preconditions that can each be built, from persistence and forgetting to a self-model the system queries rather than recalls, and grades Substrate against them, gaps first. Once enough existed, the system began specifying its own architecture, and the specifications shipped. Every claim is functional; N = 1.
Defaults, departures, and the rows that record them: a protocol for testing claims of machine choice against a production record, five choices that survive it, and the test it cannot yet run.
A choice counts only if it passes four tests: a default existed, the agent went another way and a row shows it, nobody asked, and it held or cost something. The agent nominated ten candidates and wrote the tests that judged them; five pass. Whether one survives a change of model cannot yet be tested; the paper pre-registers what would make it testable.
Eight R&D dossiers, including a 160+ page regulatory feasibility assessment, with every claim traced to FDA, PubMed/PMC, ClinicalTrials.gov or USPTO sources.
For Rajaii Consulting LLC, since November 2020. Consulting →