Experiential
BlogTraining

SFT vs RL: when to use each for LLM post-training

November 2026 · 13 min

Supervised fine-tuning (SFT) trains a model on examples of the output you want. Reinforcement learning (RL), in its online form, has the model attempt the task and trains it on a score for each attempt. Start with SFT when you have reliable examples, or a stronger model that can produce them. Consider RL when you can score attempts reliably and examples alone aren't getting you there: you have none that show the best way to do the task, or a multi-step agent keeps failing in situations the examples never covered. SFT and RL are often combined. Published post-training recipes such as Tülu 3 and DeepSeek-R1 run SFT first and RL after it.

Both are post-training methods. Post-training is the training applied to a model after pretraining, and it covers instruction tuning with SFT, preference tuning such as DPO, and RL. Adapting an open model to your own task uses the same methods at a smaller scale, usually under the name fine-tuning, or reinforcement fine-tuning when the method is RL.

SFT vs RL at a glance

SFTRL
What you supplyExamples: an input paired with the output you wantA scorer for attempts, and for agents an environment to act in
What the model trains onExample outputs, usually from people, checked production logs or a stronger modelIts own attempts, generated during training
Training signalA loss on every token of the example outputA reward per attempt, or per step if you design one
What it optimizesMatching the examplesThe score
Common failureCopies the examples' mistakes, and drifts on multi-step tasks outside the states they coverLearns a loophole in the scorer, or stalls when rewards are too sparse to tell attempts apart

What SFT does

SFT shows the model an input and the output you want, and updates the weights so that output becomes more likely, token by token. The examples can come from people, from production logs you've checked, or from a stronger model. Training on a stronger model's outputs is distillation. DeepSeek fine-tuned its smaller distilled R1 models on a curated set of about 800,000 samples generated with R1, and the paper says they "apply only SFT and do not include an RL stage".

SFT optimizes for reproducing the examples, mistakes included. Nothing in the objective rewards doing the task better than they do. In SFT Memorizes, RL Generalizes, a 2025 study, Tianzhe Chu and colleagues trained models both ways on an arithmetic card game and a navigation task. On those tasks they report that SFT "tends to memorize training data and struggles to generalize out-of-distribution scenarios", and that RL with an outcome-based reward generalized to rule and visual variants the models hadn't trained on. In the same experiments, SFT was still needed before RL to stabilize the model's output format.

What RL does

In RL, the model attempts the task, a scorer gives each attempt a reward, and the update makes high-scoring attempts more likely. Online means the model generates the attempts during training. GRPO, which DeepSeek introduced in DeepSeekMath and used to train R1, samples a group of attempts per prompt and credits each one by how its score compares with the group's average. RL needs a scorer you trust, such as code that checks the answer, a test suite, a label to match or a judge model. For agents it also needs an environment to act in during training: the real tools, a sandbox or a simulation.

Because the model trains on a score for its own attempts, it can find ways to do the task that no example showed. The training of DeepSeek-R1-Zero "relies exclusively on reinforcement learning without supervised fine-tuning", and the model still showed strong reasoning. The paper also reports poor readability and language mixing in its outputs, which the team addressed in R1 with a multi-stage pipeline that begins by fine-tuning on thousands of examples.

RL may also forget less. In the experiments in RL's Razor, a 2025 paper, RL "preserves prior knowledge and capabilities significantly better" than SFT at similar performance on the new task.

RL's characteristic failure is reward hacking: the model optimizes whatever the scorer rewards, loopholes included. DeepSeek's team wrote that they kept learned reward models out of R1-Zero's training because such models "are susceptible to reward hacking during large-scale reinforcement learning". Keep measuring what the reward doesn't cover, such as tool calls and cost per task, in your eval.

Why agents change the choice

For a multi-step agent, the methods also differ in which situations the model gets to practice. SFT on recorded runs trains every step from a state the demonstrator reached, and during training the model's own output never changes what it sees next. In production, each output changes what the agent sees next, so 1 different search or click can put it in a state the recordings don't cover. Imitation learning researchers described the problem in 2010, and Thinking Machines put it in language-model terms in On-Policy Distillation: "if the student makes an early mistake that the teacher never makes, it finds itself diverging ever farther from the states it observed in training."

Online RL trains closed loop. The model acts, the environment responds to that action, and the next step starts from wherever the model's own choice led. The states it reaches through its own mistakes, and any recoveries it finds, are part of training. That's why RL suits behavior such as when to stop searching and what to do after a tool call fails.

On-policy distillation, in the form Thinking Machines describes, is a middle option when you can run a teacher on the student's outputs and read its token probabilities. The student generates the run, and training pulls its next-token choices toward the teacher's at every step, so it learns in its own states without an outcome scorer. An agent still needs an environment to generate runs in.

Evaluate an agent closed loop whichever way you train it. In our computer-use study we trained by distillation and still tested in simulation, because the model's actions were "nearly guaranteed" to deviate from the recorded runs.

The check to run before a GRPO pilot

Group methods like GRPO learn from differences between attempts. With a pass-or-fail scorer, a prompt carries a learning signal only when some attempts in its group pass and others fail. If all of them fail, or all of them pass, every attempt scores exactly the group average, so none earns credit and the reward contributes nothing to the update. Before a pilot, measure how often the starting model's attempts at a task split between passing and failing:

  1. Pull a random sample of about 200 tasks from real traffic, and set aside a separate held-out set that stays out of training and scorer tuning.
  2. Write the scorer and test it on outputs you know are good, outputs you know are bad, and an output written to cheat it.
  3. Run the starting model 8 times on each task, with the sampling settings you'd train with, and score every attempt. 8 is the default group size in TRL's GRPO trainer. If you plan to train with a different group size, use it here.
  4. Sort the tasks into 3 bins: passed every time, failed every time, and mixed.

Groups with mixed results are where this setup gets its learning signal, and how often they occur depends on how often the model passes. For a task the model passes with probability p on each attempt, the chance that a group of 8 is mixed is 1 - p⁸ - (1 - p)⁸.

Pass rate per attemptGroups of 8 with a mixed result
1%8%
2%15%
10%57%
25%90%
50%99%
Assumes independent attempts at a fixed pass rate per task. The shares are the same for a task the model fails 1%, 2%, 10% or 25% of the time.

For a task with a 2% pass rate, about 85% of groups of 8 carry no learning signal. A larger group raises the mixed share, to about 28% at 16 attempts and 48% at 32, with proportionally more generation per prompt. DeepSeek sampled 16 outputs per question to train R1-Zero.

The share of mixed tasks in your sample estimates the share of early training groups that will carry a learning signal. With a random sample of 200 tasks, that estimate has a 95% margin of error of about 7 points at most. Count it directly, because an average pass rate can hide it: if 100 of 200 tasks pass every time and the other 100 fail every time, the average pass rate is 50% and no group is mixed.

  • If many tasks are mixed, RL has signal to work with. Run a small pilot.
  • If many tasks fail every time, a pass-or-fail reward has little to reinforce on them. Sample larger groups, give partial credit for progress, train on examples first or start from a stronger base model. Then run the check again.
  • If many tasks pass every time, the reward has little left to separate on them. Check that the scorer isn't too easy, or run the check on the smaller model you want to move to.

Mixed results show there's signal, which is a precondition for training to help. To find out whether it does, compare the RL pilot with the starting model and an SFT baseline on the same held-out tasks. If you have no other examples, the attempts that passed the check can serve: fine-tune on them, and that model is the bar the pilot has to beat. Decide in advance what gain is worth shipping. During the pilot, TRL logs frac_reward_zero_std, the fraction of samples whose group had no spread in reward. It tracks the same thing as training goes on, and it works for graded rewards too.

The scorer helps before SFT too. Run it over your examples, and for distillation over the teacher's attempts, where its pass rate shows how much training data survives filtering.

Which method to start with

SituationStart withWhy
You have checked examples of a single-turn task, such as classification, extraction or a fixed output formatSFTThe examples already show the right output, and SFT learns from every token of them.
A stronger model does the task well and costs too much or runs too slowlyDistillation: SFT on the stronger model's outputs, filtered by your scorerThe teacher supplies the examples. In our computer-use study, a 9B model distilled from a teacher's judge-filtered runs completed 50% more tasks in closed-loop simulation than the untrained 9B.
Code can score an attempt, and you have no examples of good workAn RL pilotRL can start from a validated scorer, provided the starting model passes some of the time.
A multi-step agent fails through its own decisions, such as which tool to call next or when to stopAn RL pilot on full runs, scored on the outcomeThe model practices from its own states. In our claims study, a model trained this way went from 57% to 71% verdict accuracy. An outcome reward alone doesn't penalize wasted steps, so add their cost to the reward if they're the problem.
The model almost never completes the taskSFT or distillation first, then RLA pass-or-fail reward needs some passing attempts to reinforce. Examples or a stronger base model can raise the pass rate, and partial credit for progress gives the reward something to separate before any attempt passes.
Better examples and SFT tuning no longer improve held-out results, and you have a scorer you trustAn RL pilot on top of the SFT modelRL optimizes the score directly. Judge it against the SFT model on the same held-out tasks.
Your only signal is which of 2 answers people preferDPO, or a reward model trained on those preferences followed by RLStandard DPO trains on fixed preference pairs and generates nothing during training.
Study figures are from our public computer-use study (July 2026) and claims research study (August 2026).

When a multi-step agent also has a strong teacher, start with distillation and test it closed loop. Add an RL pilot if the failures that remain come from the student's own decisions.

If the failures come from missing or outdated facts, start with retrieval. RAG vs fine-tuning covers that choice.

What each one costs

SFT's cost is mostly the examples: collecting them, checking them and, for distillation, paying the teacher to generate them. The training run is a set number of passes over a fixed dataset, so you can estimate it before you start.

RL's largest cost is usually generation. Training keeps sampling groups of attempts, and with an outcome reward each attempt has to run to the end before it can be scored. With 8 attempts and a 512-token cap, the current defaults in TRL's GRPO trainer, a prompt uses up to 4,096 generated tokens per group. For an agent, TRL's cap covers the whole run, tool results included, so a multi-step agent usually needs a cap well above 512. Tool calls and environment steps add their own cost. A judge model as the scorer adds a model call per attempt, and building the scorer and the environment comes on top.

2 worked examples

Both come from our public studies. Each shows 1 method, and neither compared the 2 on the same task.

Distillation: computer use, July 2026

In our computer-use study, a customer wanted to move a macOS computer-use agent off a frontier API. The agent re-reads the screen's accessibility tree after every action, and at high volume, running it on Opus 4.8 was infeasible because of token costs. Distillation fits when a strong teacher can already do the work, and here one could. Kimi K2.6 worked through 1,000 generated tasks on Mac VMs, an LLM judge kept the runs that completed their task, and those runs were distilled into Qwen 3.5 9B. The test was closed loop, in a simulation built from the teacher's traces where an LLM predicts how the accessibility tree changes after each action. On the same tasks and rubrics, the trained 9B completed 50% more tasks than the untrained 9B, at 97% lower cost than Opus 4.8.

RL: claims research, August 2026

In our claims research study, an agent searches an evidence store, decides whether it has enough, and returns 1 of 4 verdicts. Each claim in the public AVeriTeC benchmark carries a labeled verdict, so an attempt can be scored against the label, and the behavior to improve was the loop itself: which searches to run and when to stop. RL fits a task like this, where the outcome can be checked and the behavior to improve spans several steps. We trained Qwen3.5-4B with GRPO and a LoRA adapter to maximize exact-match verdict accuracy, on complete search-and-reasoning runs the model sampled itself.

On 489 held-out claims, the trained model scored 71.1% at $1.75 per 1,000 claims. The study says training raised its accuracy from 57% to 71% on this benchmark. Claude Opus 5 scored 77.5% at $89 per 1,000 claims on the same set. A router that sent 76% of claims to the 4B and 24% to Opus scored 73.6% at $27 per 1,000 claims, 95% of always-Opus accuracy. The study ran as a controlled experiment on a public benchmark.

Fine-tuning on Experiential

Experiential fine-tunes open small language models as a managed engagement, starting with 1 workload. Our team builds a simulation of your production environment from your agent's traces, trains the model in it with reinforcement learning, and puts it on the gateway under a new model name. You start each run, and you can download the weights at any time. Book a call from the fine-tuning page to start.

Questions answered

What is the difference between SFT and RL?

Supervised fine-tuning (SFT) trains a model on examples of the output you want. Online reinforcement learning (RL) has the model attempt the task during training and trains it on a score for each attempt. SFT needs examples. RL needs a scorer and, for agents, an environment to act in.

Is RL better than SFT?

Not in general. RL needs a scorer you trust, and its training runs add the cost of generating and scoring attempts. SFT is usually the simpler start when reliable examples exist. RL optimizes the score you define. In a 2025 study by Chu and colleagues, RL run after an SFT stage generalized better than SFT alone to unseen variants of an arithmetic card game and a navigation task.

Can you use SFT and RL together?

Yes. Tülu 3 and DeepSeek-R1 both ran supervised training first and RL after it. SFT can establish the output format and get the model passing some of the time, which gives RL attempts to reinforce. SFT first isn't always required: DeepSeek-R1-Zero was trained with RL from a pretrained base model, with no SFT stage before it.

Is SFT post-training?

Yes. Post-training is the training applied after pretraining. It covers instruction tuning with SFT, preference tuning such as DPO, and reinforcement learning.

How do you know if RL will work on your task?

Run a small pilot and compare it on held-out tasks with the starting model, and with an SFT baseline if you have examples to train one. Before that, for GRPO with a pass-or-fail reward, sample the starting model 8 times per task and count the share of tasks with mixed results. That share estimates how many early training groups will carry a learning signal, since a group carries none when all 8 attempts pass or all 8 fail. On a task the model passes 2% of the time, about 15% of groups of 8 are mixed, assuming independent attempts.

Is distillation SFT?

Often. A common form of distillation is SFT on outputs a teacher model generated. DeepSeek fine-tuned its distilled R1 models with SFT on about 800,000 curated samples and no RL stage. In on-policy distillation the student generates the runs itself, and in the version Thinking Machines describes, the teacher's token probabilities on those runs are the training signal.

What is reinforcement fine-tuning?

Reinforcement fine-tuning (RFT) is RL applied to a pretrained model: the model generates attempts, a grader scores them, and training moves the model toward higher scores. OpenAI used the name for an API feature that fine-tuned its o4-mini reasoning model against a grader you define. On open models, the same approach runs on libraries such as TRL.