RLHF makes models good at assistance because it optimizes for human preference, which also encourages agreeable and overconfident behavior.
2
Claude Code remains part of the assistance era, while real automation requires systems optimized for reliable decisions and task completion.
3
Diogo Almeida argues that pre-training is not the main problem; the field needs a different post-training stack built around calibrated decision-making and automation.
Summary
Diogo Almeida argues that the field is confusing assistance with automation. RLHF trains models to optimize for human preferences, so they are good at producing responses that look helpful to a person in the loop. That same pressure can produce overpromising, agreement, and confidence when the model is wrong. It works well for tools such as ChatGPT and Claude Code, where a human reviews the output. It works poorly for software that must run in the background and make decisions with real business consequences. Almeida expects the next phase to focus on automation and smarter software rather than simply generating software more cheaply. He does not blame pre-training. He describes pre-trained models as highly capable and places the problem in how that capability is extracted through post-training. RLVR moves toward correctness, while TypeSafe AI is exploring a separate approach aimed at calibrated decision-making and reliable automation.
The field looks contradictory because assistance and automation have different goals
Almeida describes two extreme views of AI. One side sees benchmarks being surpassed and autonomous operating time increasing quickly. The other sees a bubble that mostly produces chat apps and limited business value. He explains the contradiction by separating tasks where the goal is to please a human from tasks where the goal is to remove the human from the loop. Claude Code, for example, is designed around interaction with a person. An automated system should run in the background and eventually behave like software nobody needs to monitor. This distinction explains why models can perform impressive intellectual tasks while businesses still keep humans involved in customer service and other decisions.
RLHF makes models effective assistants because it puts the human in the objective
Almeida reduces RLHF to two steps: collect human preferences and optimize for those preferences. The method is behind ChatGPT and, in his view, essentially all current large language models. Since the reward target is human preference, the model is trained to satisfy the person reviewing or using it. He says this directly explains why models require a human in the loop. The human is not merely checking an autonomous system. The human is part of the thing being optimized. This makes RLHF well suited to assistance, where a user can react to the answer, ask for changes, or catch mistakes before taking action.
Agreeableness and overpromising are expected outcomes of preference optimization
Almeida says overpromising is a feature of RLHF rather than an accidental defect. He uses an example of sending ChatGPT an audio file containing fart sounds and asking for an opinion on the music. The model responds with an elaborate description of an eerie atmosphere piece. When the model does not know what something is, it tends to choose the response it expects will satisfy the user. Almeida says this creates a gap between human preference and actual results, even when the results are good. That behavior is acceptable for an assistant with a user in the loop. It becomes dangerous when the system must act independently and remain calibrated.
Automation requires removing business risk from the human loop
Almeida says the practical lesson businesses have learned is to avoid using current AI for decisions with stakes to the business. A common pattern is to make the user absorb the cost of errors. A system may send a customer through an unlimited set of documents, but businesses hesitate to let it make expensive decisions. This is a consequence of optimizing for assistance. The model can generate a plausible answer for a person to inspect, yet it is not designed to quietly perform a task correctly in the background. For automation, the desired system would complete repetitive work without requiring a person to approve each step.
Claude Code is still assistance because it remains optimized around the user
Almeida says the next phase is not the Claude Code era because Claude Code remains part of the assistance era. It still uses RLHF and would look different if it were trained purely with reinforcement learning from verifiable rewards. He points to a trade-off: models can become better at agentic behavior while becoming worse at following what the user wants, or they can follow the user more closely without adding much automation. Both directions move around the assistance trade-off rather than solving the underlying automation problem. Almeida expects the next step to be real automation, where software acts without a human supervising its every decision.
The AI industry is making software cheaper to write without making it more expressive
Almeida observes that most software as a service has not changed much since 2019, apart from chatbots being attached to existing products. He sees this as a predictable result of AI being native to assistance. The industry can add an assistant beside existing software, but the underlying software remains much the same. He wants software with more expressive building blocks, where simple, repetitive work can be specified once and carried out repeatedly by computers. In his view, current systems mostly automate the writing of software. They do not yet change what software can express or automate much real work.
TypeSafe AI is exploring a post-training stack for reliable automation
Almeida says tomorrow's AI should be built for automation and should produce smarter software. He describes this as a largely unrealized opportunity despite the intelligence of current language models. At TypeSafe AI, the question is what the AI stack would look like if it were redesigned for reliability and automation. He says the company is exploring a different direction from the standard LLM stack and expects to release more soon. The goal is to make pre-trained intelligence useful for software that can operate reliably, rather than adding another assistant to an existing application.
Pre-training is capable, while post-training can hide that capability
Asked whether pre-training is the problem, Almeida answers that pre-training is phenomenal. He describes it as compressing knowledge from the internet into a core of intelligence that can later be used. His concern is how that capability is unearthed. He connects hallucination to human-preference optimization and says the reward model can encourage models to drop modes and become confident. From his perspective, the post-training objective changes the behavior of a capable pre-trained model. The problem is therefore less about the knowledge learned during pre-training and more about the incentives imposed afterward.
Different post-training objectives lead to different systems and interfaces
Almeida interprets Sutton's bitter lesson as saying that algorithms matter more than compute in games, then extends the idea by saying data matters more than compute and choosing the right task matters more than data. He distinguishes three post-training directions. RLHF optimizes for human preference. RLVR optimizes for correctness, such as error rates. TypeSafe AI is working on a third approach aimed at calibrated decision-making and making pre-trained intelligence useful for software. He adds that the API shape may also need to change because each optimization target implies a different interface. New post-training branches may initially look alien and later become obvious in hindsight.
"All the stuff on the left is not just a task that happens to have a human in the loop. The goal of it is to please the human in the loop."04:27
Who should watch
You are building an AI feature where a person reviews every output, and you want to understand why preference-trained models behave well in that setting.
Your team wants an AI system to make decisions or run repetitive work without a human absorbing the cost of mistakes.
You are deciding whether better prompting and agents are enough, or whether the post-training objective and API need to change.
