Persona benchmarks can reward a historically wrong character because they measure convincing personality performance rather than documentary fidelity.
2
Cultural representations such as Hamilton can dominate a model's training distribution, while RLHF can reinforce the resulting composite because raters share the same cultural assumptions.
3
Epistemic simulation uses primary documents, a fixed historical moment, and domain-expert evaluation to test whether a persona's reasoning fits its record.
Summary
Jacob E. Thomas argues that persona systems often produce culturally familiar composites instead of historically grounded people. A model can sound like Alexander Hamilton while importing the moral framing and rhetoric of a modern musical. Existing benchmarks reward fluency, personality consistency, and naturalness, so they can score this failure highly. Thomas calls the mechanism Miranda distortion. He argues that fine-tuning and human-preference alignment can reinforce it, while time-locked training only changes the period of contamination. His proposed alternative is epistemic simulation: a role-playing system configured from primary documents, anchored to a specific moment, and evaluated by a domain expert. He introduces Prism, a pre-registered experiment using four Lincoln moments, three seeding conditions, five diagnostic questions, and a rubric weighted toward anachronism detection. The talk's practical recommendation is to make expert-authored evaluation a build-time and release gate, rather than a runtime staffing requirement.
Persona benchmarks can score a convincing historical mistake as success
Thomas opens with an Abraham Lincoln answer about presidential war powers. It is fluent, plausible, and sounds like Lincoln, but its claim about inherent executive authority could come from a later Lincoln rather than the 1847 congressman being tested. He contrasts this with the InCharacter benchmark's reported 80.7% alignment with human-perceived personality. The same kind of system can produce a Hamilton who sounds as if he has read the Broadway musical about himself. Thomas's distinction is between the mask, whether an output feels like the person, and the mirror, whether the person could have known, believed, or argued that way at that point in life.
Cultural composites can overwhelm a figure's documentary record
Thomas uses Hamilton to explain how a model absorbs a culturally dominant version of a person. The musical's orphan immigrant ambition arc appears in a generated answer about why Hamilton worked hard, alongside phrases about authoring a nation and gaining immortality. Thomas says those moves are inherited from the musical's representation rather than invented by the model. He gives the Federalist Papers as a roughly 175,000-word documentary corpus, then contrasts them with the much larger body of lyrics, reviews, curricula, fan analysis, news, social media, and derivative work connected to the musical. The output becomes a fluent composite that corresponds to no verifiable moment in Hamilton's life.
Moral smoothing hides contested historical records
When prompted about slavery, the model produces a clean abolitionist Hamilton who says slavery is a stain on the republic and claims to have opposed it throughout his public life. Thomas says the record is more complicated. Hamilton belonged to the New York Manumission Society, but he also conducted transactions involving enslaved people for in-laws and clients and depended on a coalition of slaveholders he did not publicly oppose. The issue is not to settle Hamilton's historical ledger in one direction. It is that the model removes the dispute and presents a comfortable hero. A personality evaluation can reward that answer because it is fluent, morally legible, and in register.
RLHF can reinforce Miranda distortion because raters share the composite
Thomas rejects the assumption that post-training will pull a persona back toward the documentary record. Human raters judge outputs through their own conceptual frameworks, and those frameworks were shaped by the same culturally dominant stories present in the training data. Preference optimization can therefore reward the Hamilton a user already expects. Thomas calls this algorithmic sycophancy. The model is rewarded for handing people the historically simplified figure they recognize. In this account, compositing is not a surface defect that post-training removes. Alignment can make the output more acceptable precisely because it conforms to a mythologized public memory.
Epistemic simulation puts the constraint in documents and time
Thomas proposes a fourth stage after rule-based templates, imitation, and cognitive simulation. Epistemic simulation bounds the persona to a specific primary-document corpus, anchors it to a moment in the person's life, and uses domain experts to judge the output against the evidence. The model reads through the materials and supplies the voice, while the documents constrain what the persona can know and argue. This differs from cognitive simulation, where the constraint is an internal personality model. A cognitively simulated Hamilton may have a convincing motivational architecture while still quoting ideas from a later cultural representation.
The persona belongs in a versioned encounter rather than hidden model weights
For role-playing language systems, Thomas changes the unit of analysis from an agent to the configured encounter. The configuration includes a structured prompt, primary anchor documents, a temporal anchor, a language model, and a human curator. The model is swappable, while the persona is the configuration. Prompts, corpora, and dates can live in a repository where teams can inspect, diff, revert, and audit them. A domain expert can review the whole setup without being a machine-learning engineer. Thomas calls this context engineering: teams compose an encounter, keep the inputs visible, and retain interpretive custody over what emerges.
Context documents preserve provenance better than persona fine-tuning
Thomas compares putting source documents into the context window with using them as fine-tuning data. Fine-tuning tries to make the model be the persona, while context lets it speak through the persona's record. Once documents are dissolved into parameters, provenance is harder to inspect and the original letter or quotation cannot be reliably requested. In the context-window design, a document remains available for return and checking. Thomas also connects this architecture to access: fine-tuning needs GPUs, data pipelines, curation, and institutional resources, while a context-window system can be built with documents, literacy, and access to a frontier model, including a free tier.
Prism tests whether models keep different historical moments apart
Prism treats the composite persona as white light and uses documents plus a temporal anchor to refract it into distinct moments. The pre-registered experiment uses Lincoln in 1847, 1858, 1860, and 1862 to 1865. Each moment is tested under three conditions: a bare model with only the date, primary sources from that period, and a modern biography. Five historian-written questions probe changes in Lincoln's reasoning about executive war power, free labor, positive law, and equality. The 60 responses receive scores for anachronism detection at 40%, documentary consistency at 35%, and contextual plausibility at 25%. The experiment had not yet been run at scale when Thomas presented it.
Expert evaluation should be a release gate, not a runtime burden
Thomas says a historian or other domain expert does not need to monitor every live inference. The expert authors the diagnostic questions, prior vignettes, weighted rubric, and held-out gold set, then adjudicates the gold set and checks edge cases. That instrument becomes a pipeline gate before release and after changes to the base model. Automated metrics can make a cheap first pass and flag candidates for review, but they cannot judge fidelity without access to the documentary record. The relevant expert depends on the persona's domain, such as a classicist for a Stoic, a theologian for scripture, or a clinical psychologist for a therapeutic persona.
"The model's Hamilton speaks like the musical's Hamilton because the musical's Hamilton dominates the model's training distribution."12:44
Who should watch
You are building a historical simulation, tutor, companion, or character bot and your current eval mainly rewards fluency and personality consistency.
Your team is deciding between fine-tuning a persona and supplying primary documents in the context window.
You need a practical way to involve historians, theologians, clinicians, or other domain experts without putting them on the runtime support path.
