# The Coherence Trap: Why LLMs Feel Smart (But Aren't Thinking)

Travis Frisinger | AI Engineer World's Fair 2025 | 20:47

Source: https://www.youtube.com/watch?v=u825uxb7LnA
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-coherence-trap-why-llms-feel-smart-but-arent-thinking
Published: 2025-06-03
Tags: evals, prompt-engineering, rag

## TL;DR
- LLMs feel intelligent because they reconstruct coherent patterns from distributed representations, even though they have no intent, awareness, or understanding.
- Hallucinations happen when a model completes a compelling pattern without enough factual context, so they are a consequence of coherence rather than a simple defect.
- Reliable LLM systems should use the frame, generate, judge, iterate loop, with retrieval, modular chains, and repeated context checks.

## Summary
Travis Frisinger argues that LLMs feel intelligent because their outputs are coherent, not because the models think. He describes coherence as a system property with relevance, consistency, stability, and emergence. Distributed representations and superposition let a model combine partial patterns into a new response at inference time. Prompts act like force vectors that steer this process through latent space. This explains why a model can produce useful code, writing, music, or medical patterns while still hallucinating. Frisinger treats hallucinations as predictable pattern completion when the system lacks factual anchors. His engineering advice is to build for coherence: frame the problem, generate outputs, judge them, validate where needed, and iterate. Retrieval provides contextual gravity, while modular chains reduce the damage from lost context. The talk gives engineers a practical mental model for designing prompts, agents, and evaluations without treating fluent output as evidence of thought.

## Key ideas
### GPT-4 made fluent output feel like understanding
[01:05](https://www.youtube.com/watch?v=u825uxb7LnA&t=65s)
Frisinger contrasts his first experience with GPT-3.5, whose fluency broke at edge cases and changed with small prompt variations, with GPT-4. Its words seemed to align effortlessly on the screen. He says this created a space between brittle chatbots and artificial general intelligence. Microsoft Research's paper 

### Collaborative iteration produced useful work without requiring intelligence
[02:50](https://www.youtube.com/watch?v=u825uxb7LnA&t=170s)
Frisinger describes early chat-assisted programming, which later became known as vibe coding. He used ChatGPT to build usable code, created Webcat to scrape web pages for a model without internet access, and worked with AI to build the AIBuddy.software blog. He also made a cat-metal concept album with AI-generated lyrics, music, and images. These projects convinced him that models could create coherent, useful results across different forms of work.

### Reliable interaction follows a frame, generate, judge, iterate loop
[09:22](https://www.youtube.com/watch?v=u825uxb7LnA&t=562s)
Frisinger's AI decision loop starts by framing the problem and its context. The model then generates one or more outputs. The user must judge their quality and fit instead of accepting them immediately, and may validate them against external requirements. The process then iterates. His shorter version is 

### Coherence is a system property rather than a cognitive one
[11:36](https://www.youtube.com/watch?v=u825uxb7LnA&t=696s)
Frisinger defines coherence through relevance, consistency, stability, and emergence. Outputs should stay connected to the conversation, preserve tone and terminology across turns, and remain stable when questioned or challenged. He calls coherence the infrastructure that thought navigates, rather than thought itself. This distinction explains how a model can appear to understand without having intent or awareness.

### Superposition lets models construct meaning from overlapping patterns
[14:05](https://www.youtube.com/watch?v=u825uxb7LnA&t=845s)
Frisinger uses Anthropic's research on feature superposition and concept circuits to explain why concepts are not stored in single neurons. Representations for feline, pet, and animal can overlap with representations for canine and pet. As context accumulates, the model separates relevant meaning and forms a coherent output. Meaning is therefore constructed on demand from distributed possibilities.

### Prompts steer generation through latent space
[15:00](https://www.youtube.com/watch?v=u825uxb7LnA&t=900s)
Frisinger describes a prompt as a force vector in a high-dimensional latent space. Context activates specialized patterns, such as storytelling, freedom, or household pets, and the model combines them into a response. A prompt about a detailed bird story can merge the pattern for birds with the pattern for short storytelling. The result comes from the interaction of activated subnetworks.

### Hallucinations are coherent pattern completion without factual anchors
[16:53](https://www.youtube.com/watch?v=u825uxb7LnA&t=1013s)
Frisinger says hallucinations can feel correct because the model reconstructs a compelling essence instead of checking facts. If the prompt leaves gaps, the model fills them according to its internal patterns. Retrieval-augmented generation adds factual anchors and contextual gravity, giving the model more structure to follow. In this view, hallucination is an emergent system behavior rather than a problem that can be fully removed.

### Systems should be designed for coherence, with context checks at every stage
[19:09](https://www.youtube.com/watch?v=u825uxb7LnA&t=1149s)
Frisinger divides an LLM system into the latent space, an execution layer containing tools, APIs, and retrieval, and a conversational interface that carries human intent. Prompts should be treated as system components. Long reasoning chains can lose coherence, so he recommends modular chains with reinforced context. Breaks in tone, structure, or flow can reveal where retrieval, chunking, or another tool needs adjustment.

## Notable quotes
- "LLMs aren't thinking. No awareness. No reasoning. No plan. And yet, they feel smart." (00:00)
- "Meaning isn't retrieved, it's constructed on demand from distributed sparks of possibility." (14:48)
- "Hallucinations are an indicator of coherence." (17:33)
- "LLMs are not intelligent. They're coherent. And that's their superpower." (20:20)

## Tools & references mentioned
- GPT-3.5
- GPT-4
- Microsoft Research
- Sparks of Artificial General Intelligence: Early Experiments with GPT-4
- Ethan Mollick
- Anthropic
- feature superposition
- concept circuits
- Webcat
- AIBuddy.software
- Ghost
- Cursor
- Windsurf
- retrieval-augmented generation
- AI Decision Loop
- Stanford CRFM

## Who should watch
- You are building an LLM feature and need a model for why fluent responses can still be wrong.
- Your agent or prompt workflow relies on long chains and loses context, tone, or structure along the way.
- You are evaluating model reasoning and want to separate useful coherence from evidence of understanding.

## Related talks

- [Semantic Blindness: 500,000 Sensors Confused an LLM](https://aietalks.com/talks/semantic-blindness-500-000-sensors-confused-an-llm) (Raahul Singh & Vanč Levstik, Phaidra, 16:25)
- [MCPs are Boring (or: Why We Are Losing the Sparkle of LLMs)](https://aietalks.com/talks/mcps-are-boring-or-why-we-are-losing-the-sparkle-of-llms) (Manuel Odendahl, 28:32)
- [12-Factor Agents: Patterns of Reliable LLM Applications](https://aietalks.com/talks/12-factor-agents-patterns-of-reliable-llm-applications) (Dex Horthy, HumanLayer, 17:06)
- [The Model Isn't Wrong—You're Just Bad at Prompting](https://aietalks.com/talks/the-model-isnt-wrongyoure-just-bad-at-prompting) (Dan, PromptHub, 08:54)
- [Real AI Agents Need Planning, Not Just Prompting](https://aietalks.com/talks/real-ai-agents-need-planning-not-just-prompting) (Yuval Belfer, AI21 Labs, 07:58)