On-policy self-distillation (OPSD)

Contract: teacher = the same base model, conditioned on a golden-answer demonstration; student trains to match it over its own on-policy rollouts. Caller math: This is Self-Distillation Fine-Tuning (SDFT), “Self-Distillation Enables Continual Learning” (Shenfeld et al., 2026) — the technique current survey literature groups under “on-policy self-distillation”: a single LLM acts as teacher and student under different contexts, and the student trains to match the teacher’s distribution over its own on-policy rollouts. This is SDFT’s per-token importance-sampling fallback (topk=0): advantage = teacher_logprob - student_logprob. SDFT’s primary, paper-validated mode (top-K soft-label cross_entropy, default topk=20) needs a (N, K)-shaped soft-target datum built from Whitney’s topk_prompt_logprobs sample field — a real extension, not shown here. Whitney mapping: The teacher is a frozen second Whitney run, synced once and never trained — SDFT’s own default (teacher_sync_every=None, “works comparably to EMA in our experiments”). The teacher prompt is a fixed demonstration template (question + a golden answer as an in-context demonstration), tokenized on the caller side — Whitney’s HTTP contract speaks only token IDs:

{question}

This is an example for a response to the question:
{golden_answer}

Now answer with a response of your own, including the thinking process.

Fail closed: You must supply a real tokenized demonstration prompt — there is no default golden answer.

def golden_answer_prompt_tokens() -> list[int]:
    """Your own tokenized demonstration prompt (question + golden answer)."""
    ...


def build_loss(student_sample, teacher_logprobs, cycle, request, ctx):
    del teacher_logprobs  # recomputed below against the golden-answer demo prompt instead
    prompt_tokens = request["prompt"]["token_ids"]
    teacher_prompt = golden_answer_prompt_tokens()
    data = []
    for sequence in student_sample["sequences"]:
        completion_tokens = sequence["tokens"]
        full_tokens = teacher_prompt + completion_tokens
        result = ctx.client.sampler_operation(
            ctx.teacher_run_id, ctx.teacher_sampler_id, "logprobs", {"input": {"token_ids": full_tokens}}
        )
        teacher_seq_logprobs = [float(v) for v in result["logprobs"][-len(completion_tokens):]]
        student_seq_logprobs = sequence["logprobs"]
        per_token_advantage = [
            teacher_lp - student_lp
            for student_lp, teacher_lp in zip(student_seq_logprobs, teacher_seq_logprobs, strict=True)
        ]
        data.append(rl_datum(prompt_tokens, completion_tokens, student_seq_logprobs, per_token_advantage))
    return forward_backward_payload(data, "importance_sampling")

rl_datum and forward_backward_payload are the shared datum helpers.

Get started

How it works

Cookbooks

Distillation

Coding agents

On-policy self-distillation (OPSD)