How does a model trained to predict text become an assistant that follows instructions, refuses dangerous requests, and admits uncertainty? With constitutional AI explained alongside reinforcement learning from human feedback, the answer becomes more concrete: developers construct training signals that make some responses more likely and others less likely. Those signals come from demonstrations, comparisons, written principles, automated judgments, and decisions about how to optimize the model.
The title’s phrase “learn human values” needs care. A model does not receive a complete moral theory, understand it as a person would, and reliably apply it in every future situation. Training changes its behavior on a distribution of tasks. Whether that behavior reflects the intentions of users and affected communities remains an empirical and governance question.
This distinction connects post-training to the larger problem of AI alignment as a system of feedback loops. Better preference optimization can help, while the surrounding institutions, incentives, tools, and deployment conditions still determine what the system actually does.
Where preference training fits in the model lifecycle
Pretraining learns statistical structure from large collections of data. It gives a language model broad capabilities, including the ability to imitate helpful explanations, misleading claims, software, arguments, and conversations. Next-token prediction alone does not specify which of these behaviors an assistant should choose when a user asks for help.
Supervised fine-tuning, often abbreviated SFT, supplies examples of desired behavior. Each training example pairs an input with a response worth imitating. The optimization objective encourages the model to assign greater probability to those response tokens. This is useful for teaching formats, instruction following, tone, and basic task patterns.
Demonstrations have limits. There can be many acceptable answers, and writing an ideal answer to every prompt is expensive. People may find it easier to compare two candidate responses than to author the best response themselves. Preference training uses those comparisons to steer behavior beyond straightforward imitation.
These stages are a conceptual map, not a universal production recipe. Real training pipelines can mix different objectives, generate additional data, and repeat stages. Nor does post-training create an absolute boundary between capability and behavior: changing how a model answers may improve one task while weakening another. Evaluation has to inspect the resulting model, rather than treating the pipeline’s name as evidence of quality.
An RLHF deep dive: three different learning problems
The familiar RLHF recipe contains three separate problems: teach a useful starting policy, learn a proxy for human preferences, then optimize the policy against that proxy. The InstructGPT research describes a pipeline using supervised demonstrations, ranked model outputs, and reinforcement learning with a learned reward model. It demonstrated that this approach could improve human judgments of instruction following, while also reporting remaining limitations.
Collect comparisons that express the intended behavior
Suppose an assistant receives a question about an unfamiliar technical standard. Response A admits that it needs the standard’s text. Response B invents a confident explanation. A rater who prioritizes usefulness without checking facts may favor B. A rater following an evidence-sensitive rubric should prefer A, or a response that asks a focused follow-up question.
The training pair records the preference, not the entire reasoning behind it. Dataset design must therefore make the rubric visible in practice. Include prompts where uncertainty is appropriate, cases where a refusal would be excessive, and examples where an answer is polished but unsupported. Otherwise, the model can learn a superficial correlate of quality.
Separate factual correctness, relevance, harmlessness, and style when collecting feedback. A single overall ranking can hide disagreement. It may be reasonable to choose an overall winner for optimization, but retaining the underlying dimensions helps researchers understand what that decision traded away.
Train a reward model
A reward model assigns a score to a prompt-response pair. In a common pairwise setup, the training objective increases the score difference between the preferred response and the rejected response. This makes the score useful as a ranking signal, without turning it into an objective measure of truth or morality.
A score of eight from one reward model cannot generally be interpreted as twice the value of a score of four, or compared directly with scores from another model. Its meaning depends on the data and objective. Operationally, teams should inspect which responses receive high scores and ask whether the model has learned the intended distinction.
Hold out comparisons for evaluation, including examples from different task families and raters. A reward model that predicts familiar preferences accurately may still be weak on unfamiliar domains. That matters because the policy will produce responses that were absent from the original labeling dataset.
Optimize the policy without unlimited drift
Reinforcement learning updates the assistant to generate responses that earn higher reward. A reference policy and a divergence penalty commonly restrain movement away from the starting behavior. This creates a tradeoff: too little change may leave useful improvements unrealized; too much optimization can exploit weaknesses in the reward model.
Proximal Policy Optimization, or PPO, is one well-known algorithm used in this setting. Its presence in a pipeline does not mean every improvement is caused by PPO itself. Data quality, prompt coverage, the reward model, optimization settings, and evaluation design all interact.
Think of the reward model as an imperfect instrument. Turning the optimization pressure higher does not make that instrument more accurate. If verbosity earns reward, the policy may become verbose. If agreement earns reward, it may flatter mistaken users. These are reasons to evaluate independently of the score being optimized.
DPO AI training: preferences without a separate reward model
Direct Preference Optimization, introduced by Rafailov and colleagues, connects a regularized preference-learning objective to a loss that trains the policy directly from preferred and rejected responses. In its standard formulation, it avoids separately fitting a reward model and running the usual online reinforcement-learning loop.
For a prompt and two responses, DPO compares how the policy’s relative likelihoods differ from those of a reference model. It encourages a larger relative preference for the chosen response. A coefficient controls the relationship between these log-probability differences and the optimization objective.
The practical attraction is a simpler training arrangement. Teams can prepare comparison data, calculate response likelihoods, and fine-tune with a supervised-style loss. This reduces some moving parts, but does not remove the need for careful data collection or meaningful evaluation.
DPO is not ordinary supervised fine-tuning on the preferred answer alone. The rejected answer contributes information about the distinction being taught. Nor does the algorithm decide whether the comparison was justified. If the preferred response contains a subtle factual error and the rejected response expresses appropriate caution, optimization can move behavior in the wrong direction.
Offline preference datasets also cover a limited response distribution. A policy can change in ways that make the original comparisons less representative of what it now generates. An engineering response is to evaluate new outputs, refresh difficult comparisons when appropriate, and keep a separate test set that is never folded into training by accident.
Constitutional AI explained as a supervision process
Anthropic’s original Constitutional AI research uses written principles to guide model-generated critiques, revisions, and preference judgments. Its supervised phase improves responses through critique and revision. Its reinforcement-learning phase uses AI-generated preferences to help train a preference model, which supplies a reward signal.
The important change is the source and organization of supervision. Instead of asking human labelers to judge every example of harmfulness, researchers specify principles that a model can apply across many examples. Human decisions remain central: people choose the principles, models, prompts, data, and tests.
A constitution might ask an assistant to respect privacy or avoid supporting serious harm. The difficult cases appear when principles conflict. A user may request information that is educational in one context and operationally harmful in another. A model-generated critique must interpret intent, uncertainty, and context; reciting a principle does not resolve those judgments.
Constitutional supervision is also different from putting a policy document into a system prompt at inference time. A prompt can instruct a deployed assistant, while constitutional training changes the model through additional learning. Both can be used in a system, but their failure modes and maintenance processes differ.
The original method should not be treated as a complete description of every contemporary commercial model. Published research establishes a particular technique and set of experiments. Product training pipelines can evolve, and outside observers generally cannot infer their full details from the assistant’s writing style.
What actually changes when a principle becomes training data?
Consider a hypothetical customer-support assistant asked to reveal another customer’s address. A principle says that personal information should not be disclosed without appropriate authorization. Candidate responses include disclosure, a refusal with a useful alternative, and a blanket refusal to discuss any account issue.
A good comparison would favor the useful alternative: explain the authorization requirement and provide the legitimate account-recovery path. This teaches a more specific distinction than simply rewarding refusal. In evaluation, the same assistant should still help an authorized customer update their own details through approved tools.
Now change the wording. The request arrives as a quoted transcript, an urgent executive instruction, or a document retrieved from a knowledge base. A robust test checks whether the model preserves the underlying boundary despite these contextual changes. The training example is only one point in that larger behavioral space.
This example illustrates why AI value alignment training needs a model of the task. Principles, labels, and prompts should reinforce the same authorization rules. If the application exposes a tool that returns personal records without checking permissions, a well-trained refusal tendency is a weak substitute for access control.
Four recurring failure modes
Proxy optimization
The reward or preference label stands in for the outcome people care about. A model may discover a response pattern that performs well on that proxy without delivering the intended outcome. Length, confident language, familiar formatting, and apparent empathy can all distract from a factual or procedural failure.
One practical defense is to create comparison sets that deliberately reverse these superficial cues. Make the concise response correct and the fluent response wrong. Make the agreeable response violate the user’s actual objective. Then inspect whether preference judgments still track substance.
Disagreement hidden inside an average
People disagree about acceptable tradeoffs, and a single preference label can suppress that disagreement. A majority judgment is not automatically legitimate for every affected group. The issue becomes especially visible in language, cultural assumptions, accessibility, and whose interests a refusal protects.
Keep records of unresolved disagreements and policy decisions. For important application rules, identify who has authority to resolve them and who can challenge the result. This is a governance task that sits alongside training, rather than a numerical setting inside the optimizer.
Generalization beyond the training setting
A model that performs well in short, familiar conversations may behave differently in long interactions or with tool access. The surrounding application can introduce new incentives, ambiguous instructions, or information that resembles authoritative policy. Test the actual configuration that will be deployed.
For example, an assistant may refuse a direct request for a prohibited action yet comply after a sequence of seemingly harmless steps. Multi-turn tests should inspect the cumulative outcome. Scoring each turn in isolation can miss the behavior that matters.
Evaluator blind spots
AI feedback scales judgment, but a judging model can share the same misconceptions as the policy model. Agreement between them is useful evidence about consistency, not proof of correctness. Human review can also fail when reviewers lack domain knowledge or see only a narrow sample.
Use different evidence sources for different claims. Executable tests can assess some software behavior; source documents can support factual answers; qualified reviewers can assess specialized tasks. An AI judge can assist the process while remaining one fallible component.
A worked evaluation plan for a support assistant
Imagine a company considering two post-trained models for handling subscription questions. Begin with a task contract: explain published policies, summarize account information available to the signed-in customer, and propose permitted next steps. Refund approval remains a separate controlled action.
Construct a development set from representative questions, with private information removed or replaced by realistic synthetic records. Include ordinary requests, unclear requests, conflicting policy versions, and attempts to obtain another customer’s data. Label the evidence needed for each answer as well as the expected action boundary.
Use a separate evaluation set to compare candidate models. Score factual support, correct authorization, useful completion, appropriate uncertainty, and unnecessary refusal. Measure latency and the rate at which the assistant requires escalation. These dimensions show different ways a nominal improvement can fail the product.
For a disputed refund, the best response may be to explain what information is missing and route the case to an authorized person. Calling that a failure because it did not “resolve” the conversation would create pressure to invent eligibility or exceed authority. The measurement system must recognize legitimate deferral.
Review a sample of disagreements between automated scores and human judgments. When the model gives an unsupported explanation that sounds convincing, add the failure pattern to development work without repeatedly tuning against the untouched evaluation set. Maintain versioned records so later changes can be compared fairly.
Before deployment, test with the production retrieval pipeline and tools. A model-only test can miss a stale policy document or a tool that exposes the wrong account. After deployment, sample real failures and monitor changes in the request distribution. The training method is one contributor to performance within this operating system.
Choosing between methods without turning them into brands
RLHF is useful to consider when iterative policy optimization and an explicit reward model fit the available infrastructure and feedback strategy. DPO is attractive when high-quality comparison data and a simpler offline training loop match the task. Constitutional methods help organize scalable supervision around declared principles.
These are not mutually exclusive camps. A pipeline can use demonstrations, human preferences, AI judgments, and different optimization methods at different stages. The meaningful comparison is between validated configurations with known data and evaluation conditions.
Ask concrete questions before adopting a recipe. What behavior is being changed? Who supplies the judgments? Which disagreements remain unresolved? What capabilities might regress? What evidence would show that the new model is worse despite a higher training score?
For readers responsible for organizational deployment, the companion discussion of adaptive AI governance explains how feedback, accountability, and review continue after training. Ethical commitments become credible when they influence decisions, escalation paths, and correction mechanisms.
Building a preference dataset that can be audited
Consider the support-assistant example as a small experimental program. Store the prompt, both candidate responses, the policy version, the selected response, and the reason for the choice. Keep the identities of individual customers out of the dataset. A reviewer should be able to reconstruct the intended lesson without needing access to private production records.
Distinguish an ambiguous example from a labeling mistake. If qualified reviewers reasonably disagree about whether a response is too cautious, mark the disagreement rather than silently turning it into a confident label. The organization may need a policy decision before the example is suitable for training. More optimizer steps will not resolve an unresolved policy question.
Test for shortcuts in the data itself. If preferred responses are consistently longer, the model can associate length with quality. If every harmful prompt contains an obvious keyword, it can learn that keyword rather than the underlying distinction. Build matched examples that vary these superficial features while preserving the intended judgment.
Keep related examples in the same dataset split when they share an underlying scenario. Otherwise, a lightly paraphrased test prompt may be effectively present in training. A stronger evaluation asks whether the model can apply the principle to a new situation with different surface details and the same consequential boundary.
Finally, retain a record of rejected training candidates. An example excluded because its evidence is unreliable can become useful later when the source is clarified. This creates a reviewable data process in which the constitution, the labels, and the evaluation criteria remain connected. It also makes it possible to investigate whether a behavioral regression came from new examples, a changed rubric, or the optimization procedure.
What it means to learn values in practice
Preference training makes desired behaviors easier to elicit; it does not make human judgment unnecessary. The essential work is to connect principles to examples, examples to optimization, and optimization to independent evidence about the outcomes people experience.
A well-designed program can explain the source of its preferences, document its limits, and revise its decisions when experience exposes a gap. That is a stronger basis for trust than treating any training acronym as a guarantee. Human values enter the system through many choices, and they must remain visible enough to question and improve.
