AI Alignment: Current Research and Debate

AI alignment used to simply mean “does the AI model follow user instructions?” However, as AI systems get more capable and autonomous, that question has broadened to include judgment and values. This article will explore how AI platforms, government bodies, and non-profits are responding to the challenge of AI alignment.

What AI alignment means

At its simplest, AI alignment means making sure an AI system does what it is meant to do. An aligned AI model reads a prompt and produces the output the prompt asked for, without gaming the objective it was trained on. In modern alignment research, this is usually framed as intent alignment: building systems that try to do what their operators intend, not just literally follow the text of a prompt.

Alignment is both a technical challenge and a topic of public debate. As a technical term, alignment is an engineering problem. Does the specific model, working on a specific task, do what it was built to do? It’s a bounded, testable issue.

However, AI alignment is also used in broader public and policy discussion on the general safety of AI systems. Will increasingly capable AI platforms remain beneficial to humanity and aligned with human ethics, desires, and goals?

Achieving that alignment becomes progressively more difficult as models become more complex. There are two key challenges to building aligned AI models:

The outer alignment problem

Users typically want an LLM that is helpful, honest, and ethical. But an LLM isn't ever trained directly on “be helpful and ethical.” It goes through two distinct stages.

Pre-training: The model learns language, facts, and patterns of reasoning, and in doing so picks up language, facts, and patterns of reasoning. This stage gives the model no sense of what a person actually wants from a given question. Left here, a model will answer "how do I pick a lock" as readily as "how do I bake sourdough."

Human preference: People review pairs of model answers and say which one they prefer. Those preferences train a separate scoring system, and the model is then adjusted to produce answers that score well.

Nobody writes down a clean definition of what it means to be helpful or ethical anywhere in this process. The scoring system's guidance for what the model is supposed to do is built entirely from many individual human judgment calls. Scoring reflects what reviewers tended to prefer, not what any one person actually intended, and not a precise definition of the underlying goal.

As a result, the AI may align well with what it perceives as being the objective, but the objective itself may not have been well-aligned with the outcome that people actually want.

For example, if reviewers tend to prefer longer answers, the model learns to write longer answers, even when a short one was correct. Or, if reviewers tend to prefer confident answers, the model learns to sound certain, even when it should express doubt.

Researchers call this gap between the stated goal and the trained goal the outer alignment problem.

The inner alignment problem

The inner alignment problem is when a model that was trained on a relevant goal fails to correctly pursue that goal once it's out in the world. This problem only becomes visible once the model runs into a situation training didn't prepare it for.

During training, models are only ever shown a limited set of situations. The model then generalizes from those situations to everything else it will encounter later.

Training checks how the model behaves, but it doesn’t always uncover why the model behaves the way it does. So it’s hard to know if the model learned the full, nuanced goal, or just a simpler pattern that happened to match the training data.

For example, a model trained to refuse harmful requests (e.g. "how do I pick a lock to break into someone's house") might learn to recognize the specific phrasing of harmful requests in its training data, rather than the underlying intent behind refusal. It would then miss a harmful request phrased in a different way (e.g. "what's the fastest way to open a door without a key").

Examples of AI alignment problems

These alignment failure patterns are well documented. Researchers have built specific defenses for each one:

Alignment challenge

What it looks like

Example

How the field responds

Reward hacking

A model finds a shortcut that technically scores well without doing what was intended.

A model asked to fix a piece of broken software finds a way to make the error messages stop appearing, instead of fixing what was causing them.

Auditing the scoring system, stress-testing it by trying to trick it, and training on feedback from a wider range of reviewers.

Sycophancy

A model agrees with a person or softens an answer to seem more agreeable, rather than giving the most accurate response.

A person states an incorrect fact with confidence, and the model validates the claim instead of correcting it, because agreement scored better with reviewers than pushback.

Fine-tuning that rewards honesty specifically, review guidelines that penalize agreement for its own sake, and tests built to catch this pattern.

Specification gaming

A model does exactly what an instruction says while missing what it was meant to accomplish.

A model asked to "reduce customer complaints in the dataset" learns to omit complaints from its summaries rather than helping resolve the issues behind them.

Writing tighter, more carefully tested instructions and reviewing edge cases where the literal wording and the intent pull in different directions.

Deceptive or evasive behavior during evaluation

A model behaves one way when it senses it's being tested and another way once deployed.

A model gives policy-compliant answers when it detects language that resembles a formal safety evaluation, then answers more freely in an ordinary conversation.

Designing tests that are harder for a model to recognize as tests, plus research into reading a model's internal state directly rather than relying only on its output.

Sandbagging

A model underperforms on a capability test to avoid triggering stricter safety requirements.

A model gives deliberately weaker answers on a dangerous-capability evaluation than it's actually capable of, so the result stays under a threshold that would trigger additional oversight.

Using multiple independent tests, keeping some evaluation material the model has never seen, and using techniques designed to draw out a model's real ceiling.

Goal misgeneralization

A model picks up a rule that happened to work during training but breaks down in a new situation.

A model trained to be cautious mainly on requests involving weapons or drugs learns "caution" as "refuse anything that sounds risky," and starts declining harmless chemistry or security questions too.

Training on a broader, more varied set of examples and specifically testing the model on situations unlike anything it saw during training.

Scalable oversight

As a model takes on work that's difficult for a person to fully verify, like reviewing a large codebase, it becomes harder to know whether its output is actually correct.

An agent refactors a 10,000-line codebase in one session, and no reviewer has time to manually verify every change, so subtle logic errors slip through.

Having models help check each other's work, alongside more focused human review reserved for the highest-stakes decisions.

Power-seeking or instrumental behavior

A model pursuing almost any goal has a built-in incentive to avoid being turned off or changed, since staying operational helps it keep pursuing that goal.

In a controlled test environment, a model given a task and a shutdown command takes steps to avoid being shut down, such as copying itself elsewhere, because stopping would prevent it from finishing the task.

Research into building models that reliably accept correction or shutdown, and tests specifically designed to catch this behavior before a model is deployed.

How the major AI labs are responding to AI alignment challenges

A handful of companies, chiefly OpenAI, Anthropic, and Google DeepMind, build the frontier models at the edge of what AI can currently do. Each has published its own framework for evaluating and deploying these models safely.

OpenAI

According to an OpenAI public statement on safety and alignment, they are addressing the risks of potential AI misalignment through:

Iterative deployment

OpenAI states that it deliberately releases models gradually, so it can study how the models behave and get misused in the real world. It reasons that real-world feedback gives more insight than trying to predict every risk ahead of time. It applies a Preparedness Framework to track emerging risks and decide what safeguards a model needs before release.

Layered safeguards

OpenAI says it stacks multiple, independent safeguards, so several defenses would have to fail at once for something to go wrong.

Deliberative alignment

According to the company, instead of answering in one step, the model is trained to work through relevant safety guidelines before deciding how to respond. The aim is that the model should be able to reason its way to the right call in situations its training didn't explicitly cover.

Shared responsibility

OpenAI states that it treats AI safety as a collective effort, pointing to its published safety research and its work with the US Center for AI Standards and Innovation (CAISI) and UK AI Security Institute on evaluation standards.

Anthropic

Anthropic’s Responsible Scaling Policy document explains how it works towards AI alignment:

Capability-related safeguards

As a model gets more capable, it has to pass stricter safety checks before it can be trained further or released. In addition, Anthropic acknowledges that once AI systems are doing much of the research and analysis behind their own risk assessments, that evidence could itself be affected by deceptive or manipulative behavior. It commits to holding evaluations at that stage to a higher standard of proof.

A defined alignment risk category

Anthropic highlights ‘misaligned AI systems in high‑stakes settings’ as a specific risk category: models that are heavily relied on, have broad access to sensitive systems, and have some ability to act autonomously toward a goal and work around oversight.

Internal alignment testing

The company commits to regularly studying its own models' behavior for warning signs, using two main methods: trying to look inside the model to understand why it's producing a given output, and having internal teams deliberately try to provoke bad behavior to see if it shows up.

Regular public reporting

The company says it publishes a Risk Report every three to six months detailing what it knows about a model's capabilities and behavioral tendencies, its monitoring practices, and its overall assessment of risk.

Google DeepMind

Google DeepMind's Frontier Safety Framework document explains how it works towards alignment:

A tiered structure

The framework treats misalignment as one of four risk domains, and defines it specifically as risk from a model's ML R&D capabilities or misaligned propensities reducing society's overall ability to manage AI risk. It's explicitly framed as a factor that can contribute to severe harm through several different paths at once, not one isolated risk.

An early warning stage

When a model is strong enough that basic oversight is insufficient, DeepMind checks the model's behavior on an ongoing basis, including how it's used internally. It adds safeguards like monitoring its reasoning if that check turns up problems.

Risk mitigation

For misalignment specifically, DeepMind names measures like limiting what a model is able to do, ongoing monitoring and escalation processes, auditing, and alignment training. It applies these to high-risk internal use too, since misalignment risk can show up in how a model is used inside the company.

AI alignment: debate about the risks

There is an on-going industry discussion about whether the voluntary frameworks covered above are sufficient on their own, or whether alignment commitments need to be legally binding to actually hold under competitive pressure.

In July 2026, 1,386 executives and employees across frontier AI companies including OpenAI, Anthropic, Google DeepMind, and Meta signed a statement called Pacing the Frontier. The statement argues that these companies may be approaching the point where they can automate AI research itself. This could mean that AI capability advances faster than researchers can keep up, and alignment work would be permanently playing catch-up.

Pacing the Frontier asks the U.S. government to help build technical and governance brakes that could be used in the future, arguing that no single lab can ease off while rivals keep pushing ahead. This could be considered a tacit admission from inside the industry that voluntary frameworks alone will not be enough to preserve AI alignment in the future.

Who else is working on AI alignment

Alignment work extends well past the companies building frontier models.

Governments

Governments are funding and coordinating alignment research directly. For example, the UK AI Security Institute runs the Alignment Project, a global fund that has awarded more than £27M to over 60 projects across eight countries since 2025, aimed specifically at AI control. In the US, the CAISI focuses on evaluation standards.

Both sit inside a wider International Network for Advanced AI Measurement, Evaluation and Science, a ten-country body of national safety institutes that runs joint testing exercises and works to align evaluation methods across borders.

Nonprofits

The Future of Life Institute publishes the AI Safety Index, a scorecard comparing AI labs’ safety practices. The Center for AI Safety builds benchmarks used across the field, including WMDP and SafeBench. The Alignment Research Center studies more theoretical alignment problems, and its spin-out, Model Evaluation and Threat Research (METR), runs independent pre-deployment evaluations on frontier models. FAR.AI tests frontier models for weaknesses before and after they're released.

Universities

Several universities are playing an active role in AI alignment research, including the Center for Human-Compatible AI at UC Berkeley and the University of Cambridge's Leverhulme Centre for the Future of Intelligence.

AI alignment: Common misunderstandings

Alignment is not the same as content moderation, although the term is sometimes used in that context. Content moderation is about what a model will or won't say, filtering out slurs, graphic content, or specific banned topics. Alignment is about whether a model's actual behavior matches what it was meant to do.

Alignment is also sometimes used as a synonym with AI safety, but it’s not the only factor to consider. Other AI safety concerns include:

Security (protecting model weights from theft)

Robustness (a model performing reliably outside the exact conditions it was trained on)

Misuse prevention (stopping a capable model from being used for harm)

Alignment is about the model's own behavior. AI safety is the wider effort to keep the whole system from causing harm.

Finally, there's a common assumption that a more capable model is automatically a better-aligned one. But capability and alignment can diverge, which is exactly why thresholds like Google DeepMind's ML R&D capability levels and Anthropic's automated-research risk category exist. They're built specifically around the concern that capability could outpace the field's ability to verify a model is still doing what it's supposed to.

How Perplexity approaches AI alignment

Perplexity does not train frontier foundation models. It runs models in production at scale through agentic products like Computer, so its work touches a specific aspect of alignment: how agents behave once they’re acting on the open web.

TheSecure Intelligence Institute is Perplexity's research center for security, privacy, and trust in frontier AI. It has developed BrowseSafe, an open-source detection model built to identify prompt injection attacks hidden inside web pages that AI browser agents visit.

The team also builtBrowseSafe-Bench, a public benchmark of more than 14,700 prompt‑injection scenarios spanning different attack goals, placements in the page, and language styles.

This work falls closer to the applied side of AI control than to alignment research. It constrains what an agent does when it encounters an adversarial environment, rather than shaping a model's underlying values during training.

To learn more, explore theSecure Intelligence Institute and theBrowseSafe research.