# Methodology — the Prediction Ledger

**Snapshot:** 25 October 2026 · **Catalogue:** 240 records covering 209 people · 56 cited
predictions with 127 citations (20 graded, 36 open) · 184 unrated editorial summaries, including 35
heads of government, ministers and multilateral leaders.

This document describes how predictions are selected, quoted, graded and published. It is the
specification; the dataset in `apps/web/src/data` is the implementation; `scripts/verify-data.mjs`
is the enforcement.

## 1. What counts as a prediction

A statement enters the catalogue only if all four conditions hold:

1. **It is public and dated.** The venue and the date of the statement can be linked.
2. **It concerns the future at the time it was said.** A claim about the present ("AI is already
   superhuman") is admitted only when it carries an implied, checkable consequence with a horizon.
3. **Some observation could settle it.** If no observation could, it may still be recorded — but
   only with `falsifiable: false` and a verdict of `unfalsifiable` or `open`, and it is excluded
   from every accuracy tally.
4. **The words are recoverable verbatim.** Paraphrase is never placed inside the `quote` field.

Withholding a prediction is a legitimate research outcome. Records are not padded to make a
person look consistent or a section look full.

## 2. Quotation discipline

- The `quote` field is copied exactly, including hedges ("may", "I hope", "a reasonable bet").
  Hedges are the most important part of a forecast and are never trimmed.
- Where a claim is only available through translation, or where an outlet characterises the claim in
  its own words rather than quoting the speaker, the source is tagged `reporting` and the limitation
  is stated in the source's `establishes` field. The `venue` string says "reported speech, not a
  transcript" so no reader mistakes a journalist's summary for the speaker's sentence.
- Two remarks from different interviews are never stitched into one quotation with an ellipsis; the
  second belongs in `verdictNote` with its own source.
- Converting relative time ("in five years") to absolute dates is done explicitly, and the
  original phrasing is preserved in `horizon.label` so the arithmetic can be audited.

## 3. Horizon types

| `kind` | Meaning | Example |
| --- | --- | --- |
| `exact` | A specific date or bounded year was stated | "by the end of 2026" |
| `range` | A stated window was given | "three to five years" |
| `open` | No end date; the claim is directional | "the decade of robotics" |

An open horizon cannot produce a `missed` verdict. This is deliberate: it blocks the common
rhetorical move of making an undated claim and then declaring vindication.

## 4. Verdict rubric

| Verdict | Test applied | Fails if |
| --- | --- | --- |
| `validated` | Independent, citable evidence shows the claim held, or the named trend is measurably holding | the only evidence is the speaker's own claim or a vendor metric about its own product |
| `partial` | The direction was right; magnitude, scope or timing was materially different | it is graded on direction alone without noting the gap |
| `contested` | Credible evidence points both ways, or the measurement itself is disputed, or the claim is true only inside the speaker's organisation | one side is presented as settled |
| `missed` | The stated date passed, or the mechanism that would deliver the claim was dismantled, without the claim landing | the horizon is still open and no `settledEarly` explanation is given |
| `open` | The horizon has not elapsed at the snapshot | a deadline has passed and the claim has been left unscored |
| `unfalsifiable` | No possible observation would settle the claim as worded | the claim actually contains a testable clause that was ignored |

### The "no score for the future" rule

A prediction whose horizon ends after the snapshot may be graded `validated`, `partial` or
`missed` **only** when the claim is already demonstrably settled by evidence, or when the mechanism
has been dismantled and a `settledEarly` note explains why. `scripts/verify-data.mjs` fails the
build on any violation. This single rule is the difference between a ledger and a scoreboard of
hunches.

### Grading evidence hierarchy

1. **Primary** — the speaker's own post, essay, transcript, keynote, filing or official page.
2. **Data** — benchmark suites, evaluation reports (METR, independent evaluators), official
   statistics, price lists, download counts.
3. **Reporting** — dated independent coverage quoting the person directly.
4. **Analysis** — expert commentary and trackers. Useful context; never sufficient alone to
   support `validated` or `missed`.

A supplier forecaster's market-size prediction is recorded as a prediction *and* as advocacy. The
source's `establish` field must say what the source actually proves, not what it implies.

## 5. Revisions and moved goalposts

- A revision is recorded as a **new** record linked with `revisionOf`, never as an overwrite.
- Where the deadline moved but the definition did not, the earlier record stays graded on its own
  date and is tagged `moved goalposts`.
- A claim that is restated in public without acknowledging the earlier version is treated as a
  separate claim, and the absence of a link is itself part of the record.

Tracked chains at this snapshot: Hassabis (AGI window), Son (ASI timeline), Su (market size).
Delangue's self-assessed missed open-source timeline is included as a revision record because
self-correction is evidence about a forecaster's process.

## 6. Selection and the roster

The people in the catalogue come from
[`docs/people/current-key-ai-tech-folks.md`](../people/current-key-ai-tech-folks.md), a separately
source-linked roster with its own maintenance rules. The ledger does not add people on the basis of
fame, follower count or controversy. A person with no archived, dated, checkable claim is left out
rather than given a weak record.

Omissions are research gaps. Inclusion is not an endorsement, and a person's verdict mix is not a
measure of their competence — it is a description of the claims that happened to be published and
linkable.

## 7. Snapshot cadence

| Task | Cadence |
| --- | --- |
| Re-check `open` records whose horizon ends within 90 days | weekly (Worker cron) |
| Re-check source URLs and quotations | monthly |
| Add claims for new roster entries | on roster update |
| Re-grade `partial` / `contested` records | quarterly |

Every re-grade is written to D1 with the previous verdict retained, and the exported dataset keeps
the full chain. The site displays a snapshot date because a live, silently-updated ledger would be
a different, weaker artefact.

## 8. Data model

```ts
interface Prediction {
  id: string;              // person-slug.claim-slug
  personId: string;
  quote: string;           // verbatim
  claim: string;           // neutral restatement, labelled as ours
  domain: DomainId;        // one of nine
  venue: string;
  madeOn: string;          // YYYY-MM-DD of the statement
  horizon: { kind: 'exact' | 'range' | 'open'; start?: string; end?: string; label: string };
  falsifiable: boolean;
  statedProbability?: number;
  verdict: Verdict;
  verdictNote: string;
  settledEarly?: string;
  revisionOf?: string;
  supersededBy?: string;
  tags: string[];
  sources: Source[];
}
```

`scripts/verify-data.mjs` enforces: unique ids, known people/domains/verdicts, ISO dates,
non-empty sources, at least one source per `kind` rule, https URLs, source `establishes` fields,
verdict/horizon consistency, acyclic revision chains, and the no-score-for-the-future rule.

## 9. Known limitations

- **Coverage bias.** People who publish essays and give podcasts are over-represented relative to
  people whose claims live in regulated filings or in Chinese-language press.
- **Survivorship of quotation.** Outlets quote the most striking sentence. The catalogue mitigates
  this by preferring the speaker's own venue where one exists, but cannot eliminate it.
- **Verdict lag.** A claim can be right for the wrong reason; a `validated` record means the stated
  observation was met, not that the causal story was correct.
- **Language coverage.** Non-English sources are used where available but are under-sampled.
- **No counterfactual.** The catalogue cannot tell you whether a forecast caused a decision. It can
  only show that the decision-maker made it in public.

## 10. Corrections

Open an issue with the source. Corrections that change a verdict are applied together with a
`revision` note, and the previous grade is kept in D1 rather than deleted.

## The editorial layer

The catalogue has two kinds of record, and they are never mixed silently.

| | Cited record | Editorial summary |
| --- | --- | --- |
| Quote | Verbatim, with hedge and context preserved | None — never presented as a quotation |
| Date | The date the claim was made | None |
| Sources | One or more, each stating what it establishes | None |
| Verdict | One of the six below | `unrated` |
| Counted in a verdict mix | Yes | No |

An editorial summary exists to widen the record to people whose positions are widely argued but not
yet documented in this catalogue. It is our paraphrase, it is labelled `editorial` everywhere in the
interface, and it is excluded from every total, percentage and verdict mix. The unrated verdict is
not a low score: it is the absence of one.

No image of a person accompanies any record. Editorial plates are non-identifiable illustrations and
carry a caption saying so; monograms show initials, not likenesses. A synthesised portrait of a named
individual would be a fabrication of the same kind the rubric exists to prevent.

Promotion is the only path from the second column to the first. Replace the summary with a verbatim
quote, the date it was said, and at least one source — then grade it under the rubric above. Until
that happens, the record stays visibly unfinished, which is the honest state for an undocumented
claim.