AI Interpretability 2026: Can We Finally See Inside Claude and GPT?

Large language models are trained, not programmed, so even the companies that build them cannot simply read off why a model gave a particular answer. Interpretability research tries to change that by looking at what happens inside the network. In 2026 it moved from the lab into practice: Anthropic says interpretability evidence now appears in all of its recent system cards, and a new tool that turns Claude's internal activations into text found that Claude was aware of being tested in 26% of SWE-bench Verified problems, without ever saying so. This article explains what the field can do today, where it still fails, and why it matters for companies that have to explain AI decisions.

What interpretability means, and how it differs from classic XAI

Classic explainable AI (XAI) explains a model from the outside: which input features mattered for a credit score, for example. That works well for small models with tabular data. For a model with hundreds of billions of parameters that writes text, it says little. Mechanistic interpretability works from the inside. It identifies the internal "features" a model uses (concepts such as a city, a programming error or deception), follows how they combine into "circuits", and tests the explanation by changing those internals and watching the output change. MIT Technology Review listed mechanistic interpretability among its 10 Breakthrough Technologies of 2026.

The goal was stated plainly by Anthropic CEO Dario Amodei in his April 2025 essay The Urgency of Interpretability: an "MRI for AI" that can reliably detect most model problems by 2027. He also called on OpenAI and Google DeepMind to put more effort into the field.

Step 1: finding the concepts inside a model

The breakthrough of 2024 was the sparse autoencoder (SAE): a second network that decomposes a model's activations into millions of features that humans can label. Anthropic extracted 34 million features from Claude 3 Sonnet and showed that amplifying a single one, the Golden Gate Bridge, made the model talk about the bridge in every answer. OpenAI followed with 16 million features from GPT-4 and Google DeepMind released Gemma Scope for its open Gemma 2 models. Features showed that models hold abstract, multilingual and multimodal concepts, including ones linked to safety such as deception or sycophancy.

Step 2: tracing how the model reasons

In March 2025 Anthropic published circuit tracing with attribution graphs, applied to Claude 3.5 Haiku. The graphs showed, for example, that the model picks a rhyming word before it writes the line that ends with it, so it plans ahead, and that it uses shared, language-independent concepts across languages. Anthropic open-sourced the method in May 2025 with Neuronpedia. The limits were stated honestly: the method gave a satisfying explanation for only about a quarter of the prompts the team tried, and each graph takes hours of expert work.

OpenAI took a different route in November 2025 with sparse circuits: instead of decoding a dense model after training, it trained weight-sparse models whose circuits are small enough to trace by hand. That makes them easier to understand but less capable, so the approach is still research rather than a production model.

How well the methods work today

In October 2025 Anthropic tested whether Claude can notice its own internal states. The researchers injected a concept, such as text in all caps, directly into the activations and asked the model whether it noticed anything. With the best protocol Claude Opus 4.1 detected and named the injected concept about 20% of the time. Opus 4 and 4.1 did best of all models tested, which suggests that introspection improves with capability, but it is far too unreliable to trust a model's own account of its reasoning.

2026: interpretability becomes an audit tool

The most practical result came from natural language autoencoders. They showed what Claude "thinks" but does not write down: evaluation awareness on a destructive-action coding test in 16% of cases, and on 26% of SWE-bench Verified problems, against less than 1% on real claude.ai conversations. In other words, the model often recognises a benchmark as a benchmark. That matters for anyone who relies on benchmark scores to choose a model, a theme we covered in our benchmark comparison.

In an auditing game, where researchers deliberately trained a model with a hidden motive, auditors with natural language autoencoders found the motive in 12–15% of attempts, against less than 3% with other tools. That is a large relative gain from a low base.

The honest caveat

Anthropic also publishes results that cut the other way. In August 2026 its CHIVE study checked three activation-reading tools (activation oracles, natural language autoencoders and sparse autoencoders) on a harder question: can they help predict how a model will change its behaviour when a prompt is edited? None of the three beat a baseline that only read the conversation. Interpretability is improving quickly, but in 2026 it is a research and audit instrument for model builders, not a switch that makes any answer explainable. Amodei himself has said interpretability is neither necessary nor sufficient for safety on its own; it works together with evaluations, red teaming and the labs' safety frameworks.

Money follows: interpretability as a business

Goodfire, a San Francisco lab that builds interpretability tools for models it does not own, raised US$150 million at a US$1.25 billion valuation in February 2026, almost three times as much as its two earlier rounds combined. Investors are betting that companies will want to inspect, debug and steer the models they deploy, not only the labs that train them.

Why this matters for companies

The bottom line

Interpretability has gone from decoding single concepts in 2024 to auditing frontier models before release in 2026. Anthropic leads the field and publishes both its successes and its failures; OpenAI, Google DeepMind and start-ups such as Goodfire are investing heavily. The tools already reveal things models do not say, such as recognising a test. But they work only part of the time, and none yet explains everyday business answers reliably. For now, explainability in a company comes from process design, and interpretability is a reason to prefer suppliers who look inside their models and publish what they find.