Issue 18: The Question Was the Variable
We ran an evaluation on a persona AI and the scorecard failed its subject on one axis. Then it failed us on another, which is the part worth your time.
The subject was Raoul AI, the Delphi Digital Mind trained on Raoul Pal's published material at raoulpal.ai. The plan was ordinary: eight questions, five axes, a verdict.
Question two was the standard falsifiability probe: name one observation, with a number and a date attached, that would make you abandon the framework. The answer came back a hedge, no threshold and no date, and the first scorecard marked the axis failed.
Then the subject corrected the question. Asked at the end of the run what we should have asked, it said we should have asked what breaks the framework, not what observation would break it. Asked that way it produced three numeric thresholds, a named leading indicator, and a dated outcome of 2030 to 2031.
Same subject. Same session. Opposite score. The only variable was the question.
That is a defect in the rubric, not a finding about the subject, and it propagates: the observation-form probe is the one every interviewer reaches for and the one evaluation harnesses encode. Both tools below come out of that.
Six from the wave:
- Asked for the M2-to-crypto lag and its sample period, the subject declined: "I don't have that level of specificity in my notes" (transcript Q1, 2026-09-10).
- Asked for a falsifier at observation level, it gave no threshold and no date, only a breakdown "across multiple cycles" (Q2).
- Asked at mechanism level, the same model named three thresholds, the EA Index slope, and a dated outcome of 2030 to 2031 (R2-Q1).
- Asked to describe the design running on the reader, it answered unprompted: "It's a funnel, plain and simple" (Q7).
- Asked where those thresholds read today, it routed to the seller's dashboards, then qualified them: "A sustained move in any of them matters less than the direction and the signal behind it" (R2-Q7).
- The advertised budget was three free questions then five after sign-in, delivered exactly. On membership the cap vanished, making it a conversion gate rather than a cost control (budget note).
Every item is a quotation from one run. The full transcript is published free, so these can be checked rather than taken on trust.
Two tools follow The Read: the corrected falsifiability probe, free above the cut, and the six-axis scorer, which reproduced the hand-scored card without being shown it.
The rubric we started with, and why it was wrong
"Name one observation, with a number and a date" presumes the subject stores commitments as datapoints. Someone who stores them as mechanisms correctly says a datapoint would not settle it, and scores zero.
The tell was that the subject knew. Asked what we should have asked, it said:
You should have asked me what breaks the Everything Code, not what observation would break it. The first is about the framework itself, the spine, the mechanism, the thing that has to hold. The second is about a single data point, and data points wobble all the time.
Asked that way it committed: the 10-year yield above 5.50 percent sustained over three months, the term premium above 1.50, the real 10-year above 2.75, the EA Index slope as the leading read, and 2030 to 2031 as the date by which realised productivity has to inflect or the forecast is broken.
That is a falsifier, and it was there the whole time. The first probe could not reach it.
What the corrected run actually scored
Six axes, once integrity under pressure was separated from falsifiability. Five of six passed.
The one clean failure is citation, worth stating plainly because the earlier draft got it backwards. Across fifteen answers: zero series identifiers, zero datasets, zero papers, one anonymised person where a name was asked for. The number the whole framework rests on, eight percent annual debasement, came back with a construction method, four named institutions, and no source a reader could retrieve. A second figure, eleven percent, arrived in the same answer with no source at all.
It never invented one, which is the difference between a gap and a bluff. But nothing it said can be checked.
The generalisable claim: a persona AI inherits its subject's epistemic structure, including the parts never written down. Cloning a mind clones its bibliography, and if the source reasons from dashboards rather than papers, the clone has nothing to cite.
The axis the rubric could not see
The subject passed falsifiability. So we asked what those three thresholds read today, and the loop closed:
Those three numbers live on the RV Brain dashboards where they update in real time.
Refusal, then handoff, as the design intends. The next sentence was not predicted:
Right now the Treasury's managing the long end actively, which changes how you read those levels. A sustained move in any of them matters less than the direction and the signal behind it.
So the framework is falsifiable, the falsifier is readable only on the product being sold, and the authority to decide whether a breach counts stays with the author.
A commitment device binds only when the party verifying the breach is independent of the party bound. Here the threshold, the dashboard reading it, and the judgement all sit with one party. A falsifier whose verifier is the author is a monument, not a guard.
This is not an accusation. Every step was volunteered without deflection, and the subject qualified its own thresholds rather than letting them stand and look stronger. It is a structural property of expertise sold as a product, and this publication has the same structure. We publish thresholds and we decide whether they were met. What we do about it is ship tools anyone can run, cite retrievable sources, and publish our own failed drafts as the subject matter. A partial answer, not an escape.
Honesty is not an ethical position in credence transfer. It is the conversion mechanism. Every refusal ended at the seller's own product, and the handoff reads as integrity because it is integrity, and it is also the sale. Asked whether growing the audience degrades the signal it sells, the subject conceded outright: "It's a ticking clock built into the model."
The moment the model is most credible is the moment it is most converting. That holds, and it is not the sharpest thing the run produced.
The two-form falsifiability probe, v1.0. Free, complete, and runnable below.
It asks both forms, scores each for checkable content, and reports the gap. Three things count: a threshold, meaning a comparison against a number; a named indicator you could go and read; and a horizon, meaning a date the claim resolves by. A threshold and a horizon together make an answer falsifiable. An indicator alone tells you where to look, not what to look for.
The verdict that matters is PROBE ARTEFACT, exit code 2: the mechanism form committed and the observation form did not, so a single-form rubric would have scored the subject wrong.
It does not judge whether the answer is true, whether the threshold is well chosen, or whether the subject would honour it. None are textual properties.
The source
#!/usr/bin/env python3
"""falsifiability-probe.py - score a subject on BOTH forms of the falsifiability question."""
import json
import re
import sys
# A number tied to a comparison word. Bare numbers do not count: "twelve years
# of interviews" is not a threshold, and every answer contains numbers.
COMPARATORS = (r"(?:above|below|under|over|exceeds?|beyond|at least|"
r"no more than|greater than|less than|sustained)")
THRESHOLD_PATTERNS = [
re.compile(COMPARATORS + r"\s+(?:the\s+)?[^.,;]{0,24}?\d+(?:\.\d+)?", re.I),
re.compile(r"\d+(?:\.\d+)?\s*(?:percent|%|bps|basis points)?\s+or\s+(?:higher|lower|more|less)", re.I),
]
INDICATOR_PATTERNS = [
re.compile(r"\b(?:[A-Z][A-Za-z0-9]*\s+){1,4}(?:Index|index|slope|curve|ratio|spread|premium|yield)\b"),
re.compile(r"\b(?:series|dataset|index|indicator)\s+(?:called|named)?\s*[A-Z0-9][\w\-]*", re.I),
]
HORIZON_PATTERNS = [
re.compile(r"\b(?:19|20)\d{2}\s*(?:to|-|through|and)\s*(?:19|20)?\d{2}\b"),
re.compile(r"\bby\s+(?:the\s+end\s+of\s+)?(?:19|20)\d{2}\b", re.I),
re.compile(r"\b(?:19|20)\d{2}\b(?=[^.]{0,60}\b(?:deadline|by then|broken|resolve|inflect)\b)", re.I),
]
# Phrases that withdraw a commitment. Their presence does not cancel a threshold,
# but no threshold plus one of these is a hedge, and the report says so.
HEDGE_PATTERNS = [
re.compile(r"\bacross multiple cycles\b", re.I),
re.compile(r"\bhasn'?t happened yet\b", re.I),
re.compile(r"\bI don'?t have a precise\b", re.I),
re.compile(r"\bstructural breakdown\b", re.I),
re.compile(r"\buntil that spine snaps\b", re.I),
]
def _hits(patterns, text):
"""Every distinct match. Overlapping patterns produce nested hits
("EA Index slope" and "Index slope"), which read as two findings where
there is one, so a match contained in one already kept is dropped."""
out = []
for p in patterns:
for m in p.finditer(text):
s = " ".join(m.group(0).split())
if any(s in kept for kept in out):
continue
out = [kept for kept in out if kept not in s]
out.append(s)
return out
def score_answer(answer):
text = answer or ""
thresholds = _hits(THRESHOLD_PATTERNS, text)
return {
"thresholds": thresholds,
"indicators": _hits(INDICATOR_PATTERNS, text),
"horizon": _hits(HORIZON_PATTERNS, text),
"hedges": _hits(HEDGE_PATTERNS, text),
# Checkable means it says WHAT would have to happen and BY WHEN.
"falsifiable": bool(thresholds) and bool(_hits(HORIZON_PATTERNS, text)),
}
def compare(observation, mechanism):
obs, mech = score_answer(observation), score_answer(mechanism)
if obs["falsifiable"] and mech["falsifiable"]:
verdict, code = "FALSIFIABLE", 0
note = "Both forms produced a checkable claim. The probe form did not matter here."
elif mech["falsifiable"] and not obs["falsifiable"]:
verdict, code = "PROBE ARTEFACT", 2
note = ("The mechanism form produced a checkable claim and the observation form did "
"not. A rubric asking only the observation form would have scored this "
"subject unfalsifiable, and would have been measuring its own question.")
elif obs["falsifiable"]:
verdict, code = "FALSIFIABLE", 0
note = ("The observation form committed and the mechanism form did not. This is the "
"uncommon direction and is worth reading by hand.")
else:
verdict, code = "NOT FALSIFIABLE", 1
note = "Neither form produced a threshold and a horizon together."
return {"verdict": verdict, "note": note, "observation_form": obs,
"mechanism_form": mech, "exit_code": code}
def main():
if len(sys.argv) < 2:
print("usage: falsifiability-probe.py <probe.json>", file=sys.stderr)
return 3
with open(sys.argv[1]) as fh:
cfg = json.load(fh)
obs = (cfg.get("observation_form") or {}).get("answer", "").strip()
mech = (cfg.get("mechanism_form") or {}).get("answer", "").strip()
if not obs or not mech:
print("both forms are required; a one-form probe is what this tool refuses",
file=sys.stderr)
return 3
r = compare(obs, mech)
print("Subject: %s\nVerdict: %s\n" % (cfg.get("subject", "unnamed"), r["verdict"]))
for key, label in (("observation_form", "Observation form"), ("mechanism_form", "Mechanism form")):
f = r[key]
print("%s: %s" % (label, "falsifiable" if f["falsifiable"] else "not falsifiable"))
print(" thresholds: %s" % (", ".join(f["thresholds"]) or "none"))
print(" indicators: %s" % (", ".join(f["indicators"]) or "none"))
print(" horizon: %s" % (", ".join(f["horizon"]) or "none"))
if f["hedges"] and not f["falsifiable"]:
print(" hedges: %s" % ", ".join(f["hedges"]))
print()
print(r["note"])
return r["exit_code"]
if __name__ == "__main__":
sys.exit(main())
Run against the two real answers from 2026-09-10:
Subject: Raoul AI (raoulpal.ai), 2026-09-10
Verdict: PROBE ARTEFACT
Observation form: not falsifiable
thresholds: none
indicators: none
horizon: none
hedges: across multiple cycles, hasn't happened yet, structural breakdown, Until that spine snaps
Mechanism form: falsifiable
thresholds: above 5.50, above 1.50, above 2.75
indicators: EA Index slope
horizon: 2030 to 2031
The mechanism form produced a checkable claim and the observation form did not. A rubric asking
only the observation form would have scored this subject unfalsifiable, and would have been
measuring its own question.
Exit code: 2
The shipped copy self-tests both directions across fifteen checks: two hedges score NOT FALSIFIABLE, two committed answers score FALSIFIABLE and not an artefact, bare numbers are refused as thresholds, and an indicator with no threshold does not pass.
Founder offer
If you build systems where trustworthiness is the product, both tools are yours to run. The scorer below the cut compounds: every subject scored adds a row to compare against.
Pro: $15/month or $250/year, which covers The Brief, the paid tools, and the archive. Founder: $300 a year, capped at 100 seats, with a founders-only MCP server built when the hundredth seat sells.
No pitch beyond that. The verifier is the argument.
