# Why you should care about AI interpretability Mark Bissell, Goodfire AI | AI Engineer World's Fair 2025 | 21:11 Source: https://www.youtube.com/watch?v=6AVMHZPjpTQ Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/why-you-should-care-about-ai-interpretability Published: 2025-07-27 Tags: debugging, guardrails, healthcare, multimodal ## TL;DR - Mechanistic interpretability lets engineers inspect and change model behavior by working with internal features and neurons. - Interpretability can reduce dependence on prompt edits, judges, and fine-tuning by giving developers tools for attribution, steering, dynamic prompting, and model diffs. - The same access to model internals can create new interfaces for image generation and help researchers extract biological knowledge from genomics models. ## Summary Mark Bissell explains mechanistic interpretability as reverse engineering neural networks to understand their internal features. He argues that the field now matters to AI engineers because it is moving beyond research demos into tools for building and debugging deployed systems. Goodfire's Ember platform lets developers inspect which features contributed to a token, turn features up or down, and conditionally change prompts during generation. Bissell also demonstrates Paint with Ember, an image interface where users place learned concepts on a canvas and adjust their strength. Other applications include guardrails, model diffs after training, explainable outputs, model pruning, and extracting scientific knowledge from genomics systems. He is candid that current methods are still developing. Sparse autoencoders are a current way to find features, but new techniques may replace or supplement them. His final reason for caring is simple: engineers want to understand how the systems they build work, and these models remain largely unexplained. ## Key ideas ### Interpretability means reverse engineering a neural network's internal concepts [00:58](https://www.youtube.com/watch?v=6AVMHZPjpTQ&t=58s) Bissell defines interpretability, also called mechanistic interpretability, as reverse engineering neural networks to understand what happens inside them. He uses the analogy of opening a black box or performing brain scans on a model. In Anthropic's Golden Gate Claude demonstration, researchers found internal features associated with the Golden Gate Bridge. By forcing those features to stay active, they made a version of Claude that brought up the bridge regardless of the subject. The example shows that interpretability can do more than explain an output. It can expose a model's internal representation and then use that representation to change the model's behavior. ### Interpretability can give developers finer control than prompt editing [03:02](https://www.youtube.com/watch?v=6AVMHZPjpTQ&t=182s) Bissell describes a common development loop: an agent fails an evaluation, the team edits the system prompt, and the edit fixes one behavior while causing another to fail. An LLM judge adds cost and another system to maintain. Fine-tuning requires domain-specific data and can produce spurious correlations, mode collapse, or reward hacking. Goodfire's Ember is intended to provide a more direct way to debug and program models at the feature or neuron level. In his example, a Llama model revealed features related to sensitive information when generating a response about confidentiality. Increasing the strength of that feature made the model refuse to reveal the email address. ### Dynamic prompting can react to features during generation [09:30](https://www.youtube.com/watch?v=6AVMHZPjpTQ&t=570s) Ember can watch for an internal feature and intervene while a model is generating. Bissell calls this dynamic prompting. If a feature related to beverages and consumer brands activates, the system can inject a different prompt telling the model that it is an assistant for Coca-Cola. In the demonstration, a question about drinks to pair with pizza triggers the feature, and the output shifts from a generic recommendation toward Coca-Cola. The user sees one continuous response and does not see the internal conditional. Bissell says the same type of neural programming can support jailbreak resistance, conditional information lookup, and other guardrail behaviors. ### Interpretability also applies to training and production monitoring [11:21](https://www.youtube.com/watch?v=6AVMHZPjpTQ&t=681s) Bissell says Ember is already being used by Rakuten for multilingual personally identifiable information detection in a chatbot and by Hayes Labs for red teaming and guardrail work. He also describes model diffs as an active research direction. After post-training, developers could compare which internal features changed, much like using a git diff. A team might detect that a model had become sycophantic before deployment by examining the features most affected by training. This approach could provide a more direct view of what training changed than relying only on input-output evaluations. ### Image models can become interactive canvases instead of prompt boxes [12:41](https://www.youtube.com/watch?v=6AVMHZPjpTQ&t=761s) In the Paint with Ember demonstration, users work with concepts learned by an image model directly on a canvas. Bissell paints a pyramid structure and a wave, moves them around, erases them, and adds a lion face. The interface connects canvas locations to the model's internal neurons, so the user specifies where concepts should appear. Users can also paint actions, such as an open mouth, and adjust their strength. Turning the feature down makes the lion open its mouth less, while turning it up makes the lion roar. This gives users familiar controls for manipulating generated images without relying only on text prompts. ### Feature combinations expose how an image model organizes concepts [15:01](https://www.youtube.com/watch?v=6AVMHZPjpTQ&t=901s) Paint with Ember lets users inspect sub-features inside a broader concept and interpolate between them. Bissell adjusts sub-features of a lion face and moves toward a more rat-like creature. He then subtracts the main lion feature and varies other components, producing something more tiger-like. He describes this as a hint about how the model may organize concepts in a high-dimensional vector space, with a rough relationship such as tiger being lion minus mane. The interface therefore provides both a control surface and a partial view into the model's internal representations. ### Interpretability could help extract scientific knowledge from powerful models [16:22](https://www.youtube.com/watch?v=6AVMHZPjpTQ&t=982s) Bissell points to explainable outputs for regulated fields such as finance, healthcare, and law. He also discusses work with the ARC Institute on EVO 2, a foundational genomics model that predicts genomic information for many organisms. The aim is to discover biological concepts the model learned that humans do not yet know and make those concepts useful to domain experts. In work with a major health system, interpretability is being applied to genomics models that estimate disease likelihood and treatment response. High performance alone is not enough for this use. Researchers also want to understand the principles behind the predictions and use them to update knowledge in the field. ### Interpretability may reduce model size and wasted computation [18:00](https://www.youtube.com/watch?v=6AVMHZPjpTQ&t=1080s) Bissell says internal analysis could show which weights are spent memorizing information that a model does not need. Developers might then reuse those weights for more useful tasks or prune parts of the model. He gives the example of creating a Claude version focused only on coding and removing parameters that are unnecessary for that purpose. This is presented as a possible efficiency and speed benefit rather than a finished method. The same access to internal features could inform choices about what a specialized model needs to retain. ### Engineers have a reason to care because these systems remain unexplained [18:35](https://www.youtube.com/watch?v=6AVMHZPjpTQ&t=1115s) Bissell closes with a motivation that is separate from the product demos. He says engineers like to understand how systems work, take them apart, and ask why they behave as they do. He finds it frustrating that current models can perform useful tasks while their internal processes remain largely unknown. That lack of understanding is also what makes interpretability interesting to him. He recommends that people who want to study examples such as Golden Gate Claude look into sparse autoencoders, while keeping in mind that the field may develop substantially different methods for finding features over the next few years. ## Notable quotes - "Interpretability is really all about reverse engineering neural networks to understand what is going on inside of them." (00:58) - "What if you could debug and program your models at the neuron level to get more of those guarantees that we're used to with traditional software development?" (06:44) - "This canvas is plugging directly into the internal neurons of the model." (13:37) - "The hallmark of an engineer is that we like to understand how systems work." (18:54) - "The current best practice way to find these features is through the use of an interpreter model called a sparse autoencoder." (20:01) ## Tools & references mentioned - Goodfire AI - Ember - Anthropic - Golden Gate Claude - Claude - Llama - Rakuten - Hayes Labs - ARC Institute - EVO 2 - Coca-Cola - Palantir - sparse autoencoders ## Who should watch - You are building an LLM agent or pipeline and keep fixing one evaluation failure with prompt edits while creating another. - You need model behavior controls or guardrails that can work during generation, rather than relying only on prompts, judges, or fine-tuning. - You work with image or scientific models and want direct interfaces or explanations tied to the model's learned concepts. ## Related talks - [Form factors for your new AI coworkers](https://aietalks.com/talks/form-factors-for-your-new-ai-coworkers) (Craig Wattrus, Flatfile, 15:35) - [Second Order Effects of AI](https://aietalks.com/talks/second-order-effects-of-ai) (Cheng Lou, 21:46) - [Build AI Systems for Discernment, Not Approval](https://aietalks.com/talks/build-ai-systems-for-discernment-not-approval) (Angel Ortmann Lee, Duolingo, 25:53) - [AI and Human Whiteboarding Partnership](https://aietalks.com/talks/ai-and-human-whiteboarding-partnership) (Christopher Chedeau, Excalidraw, 16:59) - [Open Questions for AI Engineering](https://aietalks.com/talks/open-questions-for-ai-engineering) (Simon Willison, Independent open source developer, 24:33)