Is AI easier to understand than scientists thought?

Scientists have spent years trying to figure out what happens inside AI systems, but the answer may be simpler than they thought.

A provocative research essay by Nick Cammarata of AI research company Goodfire has reopened a debate about how artificial intelligence processes information. Cammarata argues that individual artificial neurons, the mathematical components that help AI systems recognize patterns, may have much simpler functions than researchers assumed.

His preliminary findings challenge a theory that has guided years of research, although experts caution that understanding individual neurons does not necessarily explain how an entire AI system makes decisions.

“Assigning a label to every neuron in a model is not the same thing as understanding the entire model,” Jacob Andreas, an MIT Associate Professor who studies AI interpretability, told IBM Think in an interview.

For years, researchers studying AI interpretability—the effort to understand how neural networks operate—have struggled to explain why individual artificial neurons sometimes respond to seemingly unrelated concepts.

One influential explanation, known as superposition, holds that neural networks pack multiple concepts into the same components to use their limited capacity efficiently. Scientists developed tools called sparse autoencoders to separate those overlapping representations into distinct features they could study.

In his , Neural Networks Are Already Clean, Cammarata argues that researchers, including himself, may have misinterpreted some of the evidence for superposition. Rather than combining unrelated concepts, neurons may recognize patterns that share connections researchers initially overlooked.

Using analytical tools and AI agents, Cammarata examined individual neurons across several neural networks. In a preliminary analysis of Qwen3-32B, a language model with 32 billion parameters, he estimated that roughly 90% of neuron poles had one main function. A neuron pole refers to the positive or negative side of a neuron’s activity.

About 7% appeared to have two related functions.

Cammarata acknowledged that these estimates depend partly on how researchers classify weaker neuron activations. He also described the essay as a “working hypothesis” rather than a completed technical study.

Outside researchers have questioned whether identifying a neuron’s strongest responses reveals enough about its behavior under ordinary conditions.

“The bigger issue (or missing part in this study) is that the peaks cover a very small fraction of a neuron’s firings,” A.B. Siddique, an Assistant Professor at the University of Kentucky, told IBM Think.

A neuron may respond to a recognizable concept when its activity peaks but behave differently during the much more frequent periods when its activity remains low, Siddique explained.

The essay offers examples of how existing tools can misidentify neurons. One neuron appeared to respond to commas but actually tracked references to Colorado. Another seemed to recognize both Bengali text and Base64-encoded strings, a method of representing digital data as text, until closer investigation revealed a shared linguistic pattern.

Siddique said researchers need experiments that deliberately change neuron activity and test whether models respond as their explanations predict. Such tests could help establish whether a neuron truly performs one function.

The findings also raise questions about whether scientists still need the sparse autoencoders they developed to separate overlapping information.

Georg Lange, a researcher who studies how neural networks work, told IBM Think that dictionary learning could still be useful. The technique helps researchers identify and understand patterns in neural networks. While Cammarata focused on neurons when they were most active, dictionary learning could help explain what happens when their activity is weaker.

Even direct experiments face obstacles. Lange noted that large networks contain redundant components and can sometimes compensate when researchers interfere with individual neurons.

For Michael Hahn, a researcher studying language models, another uncertainty remains: whether explanations that make sense to people capture everything neurons actually do.

“There is nice evidence here that individual neurons are often correlated with interpretable semantic concepts,” Hahn told IBM Think. “A huge challenge here is that we don’t know if these interpretations really capture everything about the neurons’ functions, or just the human-understandable component of a potentially much more complicated semantics.”

Beyond the scientific debate lies a practical concern for companies developing AI systems: Would understanding individual neurons help researchers predict dangerous behavior?

At IBM Research, Pin-Yu Chen, a Principal Research Scientist working on AI safety and security, questioned the strength of Cammarata’s conclusions.

“I do have mixed feelings about that essay because I think it’s a bit overhyped,” Chen told IBM Think.

Even with a detailed understanding of a model’s internal components, researchers would still face the challenge of determining how those components interact, Chen said. He compared an understandable neural network to a maze with many possible paths that still require exploration.

“Having a map of the maze can make red teaming more efficient, but having a map alone without an efficient search tool wouldn’t make safety complete,” he said.

Red teaming is an offensive security practice that involves deliberately testing an AI system to uncover weaknesses or potentially harmful behavior.

Chen pointed to IBM’s research on automated safety testing and its open-source vLLM-Hook project, which allows developers to inspect and modify internal model activity as an AI system generates responses. Most interpretability research examines models outside normal operation, he said, making it difficult to turn those findings into safeguards that work while a model runs.

Research at the University of Maryland, Baltimore County, illustrates the difficulty. Computer scientist Manas Gaur described experiments in which changing combinations of AI features sometimes made model responses more harmful.

His team’s findings suggested that apparently unrelated features could interact in ways their individual descriptions did not predict. The effects varied across models, prompts and experimental conditions. Gaur also cautioned that the 90% figure, which Cammarata identified as the volume of neurons with one job, may depend on the prompts used, the model’s size and the context in which neurons are tested. “Further testing is needed to establish how broadly it holds across multiple scenarios,” he told IBM Think.

If independent experiments confirm his findings, Cammarata’s hypothesis could offer researchers a more direct way to study neural networks. But whether knowledge of individual components would help scientists anticipate the behavior of entire models remains uncertain.

For Hahn, understanding how neurons work together matters more than assigning each one a recognizable function.

“As long as we don’t understand the reasoning processes, we can’t anticipate the range of things that could happen inside language models, and we can’t anticipate the things that could go wrong,” he said.