More algorithms

Every algorithm on this page extends the GRPO primitive spine unless noted. Your code owns the math; Whitney owns ordered execution. Every entry has a real, runnable file under examples/training/http/ — either a build_loss callable plugged into rl_loop.py or teacher_loop.py, or (for sapo, the one entry the provider itself doesn’t expose) a non-allocating capability-index stub that explains exactly what is missing. See Runnable reference implementations for the full table. ⚠️ loss_config reaching the provider at all is a recent fix. Whitney’s Modal trainer bridge (training/runtime/modal_unified_trainer.py) used to call the provider’s forward_backward/forward with only (data, loss_function) — silently dropping every caller’s loss_config, so PPO’s clip range, DPPO’s divergence thresholds, and similar per-loss tuning always ran on the provider’s own defaults regardless of what a caller set. That’s now threaded through; confirm it with a live smoke (two otherwise-identical requests with different loss_config values producing different results) before trusting a non-default value in production.

Shared Whitney loop

GET /v1/training/capabilities
  → POST /v1/training/sessions
  → POST /v1/training/sessions/{id}/runs
  → repeat per cycle:
      save_weights_for_sampler(N) → sample(N) → [optional logprobs]
      → build_loss(sample, cycle, request, ctx) -> ForwardRequest
      → forward_backward → optim_step
  → save_state / export_lora → finish → close

Required RL operations for most entries: forward_backward, optim_step, save_weights_for_sampler, sample, logprobs. Before allocating GPU work, confirm the capability row for your provider and model advertises every operation you use. If the contract cannot be represented in Whitney JSON primitives, stop before creating a session. GET /v1/training/capabilities does not return a loss-function list — it cannot, because Whitney forwards loss_function straight through to the provider. examples/training/http/whitney_datums.py’s PROVIDER_LOSS_SUPPORT/require_advertised_loss is a hand-maintained, caller-side table of what each provider actually accepts (cross_entropy/importance_sampling/ppo/cispo on both; dro Tinker-only; gspo/ppo_critic/dppo Modal-only) — a hint that saves a wasted allocation, not a live capability.

GRPO without standard deviation

Contract: group advantage without standard-deviation normalization. Caller math: After sampling, compute group-mean advantages only. Do not divide by per-group standard deviation. Whitney mapping: Same loop as GRPO. Encode the no-std objective in ForwardRequest.loss_function and loss_config. Fail closed: Do not use generic GRPO with std normalization. Code: grpo_nostd.py — group_mean_advantages(rewards, normalize_std=False), upstream tinker_cookbook’s own default.

advantages = group_mean_advantages(rewards, normalize_std=False)

Length-normalized GRPO

Contract: explicit token or sequence length-normalized loss. Caller math: Apply your documented length weighting when building the loss tensor inputs inside ForwardRequest. Whitney mapping: Standard GRPO loop; length normalization lives entirely in caller-constructed loss_inputs and loss_config. Fail closed: Whitney does not apply length normalization implicitly. Code: grpo_len_norm.py — divide each trajectory’s scalar advantage by its own completion length before padding.

advantages = group_mean_advantages(rewards, normalize_std=True)
normalized = [length_normalize(a, len(seq["tokens"])) for a, seq in zip(advantages, sequences)]

Dr.GRPO

Contract: no-std, length-normalized GRPO with no implicit KL penalty. Caller math: Combine no-std group advantages, explicit length weighting, and omit any implicit KL term your generic GRPO helper might add. Whitney mapping: Standard GRPO loop with a custom ForwardRequest. Fail closed: Not interchangeable with generic GRPO or length-normalized GRPO alone. Code: drgrpo.py — composes grpo_nostd.py and grpo_len_norm.py; “no implicit KL” needs no extra code since neither rl_loop.py nor whitney_datums.rl_datum ever adds one on their own.

advantages = group_mean_advantages(rewards, normalize_std=False)
normalized = [length_normalize(a, len(seq["tokens"])) for a, seq in zip(advantages, sequences)]

PPO

Contract: policy plus first-class critic/value network and GAE lifecycle. Caller math: Group-relative advantages, same as GRPO — no critic. Reading the pinned Modal/SkyRL source settles this: ppo_policy_loss reads (log_probs, old_log_probs, advantages, config, loss_mask, rollout_logprobs), the same four tensors as importance_sampling. The critic-needing variant is the separately advertised ppo_critic, not plain ppo. Whitney mapping: ppo is a real loss name on both providers. The clip range is set two different ways per provider, same key names:

Tinker Cloud: clip_low_threshold/clip_high_threshold as per-token tensors (rl_datum(..., clip_low=, clip_high=)).
Modal: the same two names as scalar loss_fn_config entries (skyrl/tinker/api.py’s _ALLOWED_KEYS_BY_LOSS_FN) — see the loss_config note above.

Fail closed: ppo.py gates with require_advertised_loss(ctx.provider, "ppo") and never sends ppo_critic.

tinker_clip = ctx.provider == "tinker"
data = [
    rl_datum(
        prompt_tokens, seq["tokens"], seq["logprobs"], adv,
        clip_low=PPO_CLIP_LOW if tinker_clip else None,
        clip_high=PPO_CLIP_HIGH if tinker_clip else None,
    )
    for seq, adv in zip(sequences, advantages)
]
loss_config = {} if tinker_clip else {"clip_low_threshold": PPO_CLIP_LOW, "clip_high_threshold": PPO_CLIP_HIGH}
return forward_backward_payload(data, "ppo", loss_config)

CISPO

Contract: provider-advertised CISPO importance-sampling loss and data layout. Caller math: Build standard GRPO-shaped data (group-mean advantages over rl_datum); the only difference from generic GRPO is loss_function. Whitney mapping: cispo is in both providers’ advertised loss sets. On Modal, clip_low_threshold/clip_high_threshold are valid scalar loss_config keys (same mechanism as PPO, above); Tinker Cloud’s CISPO loss_config contract isn’t published anywhere reachable from this repo, so cispo.py leaves it empty there. Fail closed: Never fall back to generic importance sampling or GRPO — set loss_function explicitly. Code: cispo.py — gated with require_advertised_loss(ctx.provider, "cispo") before building the payload.

require_advertised_loss(ctx.provider, "cispo")
data = [rl_datum(prompt_tokens, seq["tokens"], seq["logprobs"], adv) for seq, adv in zip(sequences, advantages)]
loss_config = {"clip_low_threshold": CISPO_CLIP_LOW, "clip_high_threshold": CISPO_CLIP_HIGH} if ctx.provider == "modal" else {}
return forward_backward_payload(data, "cispo", loss_config)

GSPO

Contract: provider-advertised GSPO loss and data contract. Caller math: Same group-relative, no-critic shape as PPO/CISPO — GSPO’s “sequence-level importance ratios” are computed by the provider from the same weights/logprobs tensors; nothing extra to build caller-side. Whitney mapping: gspo is a real, Modal-only loss name — absent from Tinker Cloud’s LossFnType entirely. Its recommended sequence_mean reduction mode is not a settable loss_config key on the pinned SkyRL SHA (only clip_low_threshold/clip_high_threshold are), so it runs on whatever Whitney’s deployed default uses. Fail closed: gspo.py gates to Modal only via require_advertised_loss. Not a generic importance-sampling fallback.

DPPO

Contract: DPPO plus critic/value lifecycle. Caller math: No critic needed — dppo_policy_loss replaces PPO’s ratio clipping with a divergence-based binary mask computed from rollout_logprobs, reading the same tensor set as every other RL loss here. Whitney mapping: dppo is a real, Modal-only loss name. Its divergence variant (binary_tv vs binary_kl) is fixed by Whitney’s deployed default — not a settable loss_config key; only the divergence thresholds (delta_low/delta_high) are. Fail closed: dppo.py gates to Modal only. Absent from Tinker Cloud’s LossFnType entirely.

loss_config = {"delta_low": DPPO_DELTA_LOW, "delta_high": DPPO_DELTA_HIGH}
return forward_backward_payload(data, "dppo", loss_config)

DAPO

Contract: dual clipping, dynamic rejection or resampling, overlong-sample filtering. Caller math: After sampling, filter overlong or invalid outputs, resample per your declared policy, compute advantages on the retained group, apply upper and lower clipping, then build ForwardRequest. Whitney mapping: Multiple sample cycles may be needed per optimizer step. build_loss receives a LossContext with a live client handle, so it can issue additional sample operations itself, inside one cycle, rather than needing a separate loop file. Each resample is still a real, billed, ordered operation. Neither provider exposes a dedicated “dual-clip” loss name on the wire — only regular/ppo-shaped clipping is selectable — so this approximates DAPO with loss_function="ppo" plus a clip range, the same per-provider mechanism as the PPO section above, not the paper’s asymmetric dual clip. Fail closed: Filtering and resampling semantics must be caller-visible; do not pretend one generic sample batch is DAPO. Code: dapo.py:

def is_overlong(sequence):
    return sequence.get("stop_reason") == "length"

retained = [s for s in sequences if not is_overlong(s)]
while len(retained) < MIN_GROUP_SIZE:
    resampled = ctx.client.sampler_operation(ctx.run_id, ctx.sampler_id, "sample", request)
    sequences += resampled["sequences"]
    retained = [s for s in sequences if not is_overlong(s)]
tinker_clip = ctx.provider == "tinker"
data = [
    rl_datum(
        prompt_tokens, s["tokens"], s["logprobs"], adv,
        clip_low=DAPO_CLIP_LOW if tinker_clip else None,
        clip_high=DAPO_CLIP_HIGH if tinker_clip else None,
    )
    for s, adv in zip(retained, advantages)
]
loss_config = {} if tinker_clip else {"clip_low_threshold": DAPO_CLIP_LOW, "clip_high_threshold": DAPO_CLIP_HIGH}
return forward_backward_payload(data, "ppo", loss_config)

SAPO

Contract: provider-native SAPO loss with explicit tau parameters. Caller math: N/A — this is the one entry in the index that cannot be sent today, not a caller-side gap. Whitney mapping: The pinned SkyRL SHA does have a real, working sapo_policy_loss internally — but Whitney’s Modal adapter calls through SkyRL’s Tinker-compatible API server, whose request model hard-gates the selectable loss to a fixed Literal of seven names. "sapo" isn’t one of them: a loss_fn="sapo" request is rejected by SkyRL’s own validation before it ever reaches the registry that supports it. Fail closed: sapo.py stays a stub — not because the caller-side contract is unclear, but because the provider genuinely does not expose this loss on the pinned SHA. Do not substitute a generic clipping loss.

RLOO

Contract: leave-one-out baseline plus reference or KL logprobs. Caller math: Compute leave-one-out baselines across grouped samples. Fetch reference logprobs via logprobs when KL is part of the objective. Whitney mapping: sample → logprobs (reference) → forward_backward with RLOO-encoded loss. Fail closed: Group-mean GRPO is not a valid RLOO substitute. Code: rloo.py — the leave-one-out baseline only; the optional reference-KL term is not included (see REINFORCE++ for that pattern via teacher_loop.py).

advantages = leave_one_out_baseline(rewards)  # reward[i] - mean(rewards excluding i)

REINFORCE++

Contract: REINFORCE++ with K2 and reference-KL semantics. Caller math: Implement K2 baseline and reference-KL terms in caller code. Use logprobs for reference policy probabilities. Whitney mapping: A single run’s sampler only ever serves its current weights, so a genuine frozen reference needs a second Whitney run — teacher_loop.py creates one, syncs it once, and never trains it, then supplies its logprobs on the student’s own generated tokens to build_loss every cycle. Fail closed: Requires advertised K2/reference-KL semantics. Code: reinforce_pp.py — the K2 KL estimator, 0.5 * (student_logprob - reference_logprob) ** 2, is one reasonable reading of “K2 semantics”, not a literature citation:

per_token_advantage = [
    group_advantage - KL_COEF * 0.5 * (student_lp - reference_lp) ** 2
    for student_lp, reference_lp in zip(student_logprobs, reference_logprobs)
]

MaxRL

Contract: explicit MaxRL estimator. Caller math: Encode the MaxRL objective directly in ForwardRequest. No provider configuration name substitutes for this contract. Whitney mapping: Standard RL operations; estimator logic is entirely caller-owned. Fail closed: Do not map MaxRL to generic GRPO. Code: maxrl.py — no canonical MaxRL formula exists; this is one reasonable, clearly-editable choice (a leave-one-out “beat the group’s best” estimator), not a spec.

def max_relative_advantage(rewards):
    return [r - max(rewards[:i] + rewards[i + 1 :]) for i, r in enumerate(rewards)]

On-policy distillation (OPD)

Contract: separate teacher or reference client and isolated teacher-data exchange. Caller math: Run teacher inference outside the training run (or via a separate sampler) and pass teacher logits or targets into your distillation loss. Whitney mapping: teacher_loop.py creates a second Whitney run for the teacher (any model — pass --teacher-model for a different, typically larger, model), syncs its sampler once, and never trains or samples it: it only answers logprobs on the student’s own generated tokens. There is no external reward — the signal is purely how much the teacher would have preferred the student’s choices. Fail closed: Do not mix teacher and student weight versions on the same sampler without explicit versioning. Code: opd.py — port of tinker_cookbook/distillation/train_on_policy.py’s incorporate_kl_penalty: reverse KL as log p - log q, i.e. advantage = teacher_logprob - student_logprob, per completion token.

per_token_advantage = [
    teacher_lp - student_lp for student_lp, teacher_lp in zip(student_logprobs, teacher_logprobs)
]

On-policy self-distillation (OPSD)

Contract: teacher = the same base model, conditioned on a golden-answer demonstration; student trains to match it over its own on-policy rollouts. Caller math: Port of tinker_cookbook/distillation/sdft.py — Self-Distillation Fine-Tuning (SDFT), “Self-Distillation Enables Continual Learning” (Shenfeld et al., 2026) — the technique current survey literature groups under “on-policy self-distillation”. This ports SDFT’s per-token importance-sampling fallback (Config.topk=0): advantage = teacher_logprob - student_logprob. SDFT’s primary, paper-validated mode (top-K soft-label cross_entropy, default topk=20) needs a (N, K)-shaped soft-target datum from Whitney’s topk_prompt_logprobs sample field — a real extension, not implemented here. Whitney mapping: teacher_loop.py’s second Whitney run, synced once and never trained — SDFT’s own default (teacher_sync_every=None, “works comparably to EMA in our experiments”). The teacher prompt is sdft.py’s own DEFAULT_DEMO_TEMPLATE (question + a golden answer as an in-context demonstration), tokenized on the caller side — Whitney’s HTTP contract speaks only token IDs. Fail closed: opsd.py — golden_answer_prompt_tokens() is a placeholder that raises until you supply a real tokenized demonstration prompt.

teacher_prompt = golden_answer_prompt_tokens()  # tokenized DEFAULT_DEMO_TEMPLATE
for sequence in sequences:
    full_tokens = teacher_prompt + sequence["tokens"]
    result = ctx.client.sampler_operation(ctx.teacher_run_id, ctx.teacher_sampler_id, "logprobs", {"input": {"token_ids": full_tokens}})
    teacher_lps = result["logprobs"][-len(sequence["tokens"]):]
    advantage = [t - s for s, t in zip(sequence["logprobs"], teacher_lps)]

SDPO (Self-Distillation Policy Optimization)

Contract: teacher = the same base model, conditioned on feedback about the student’s own prior attempt (not a full reference answer — that’s OPSD). Caller math: Same teacher-forced mechanism as OPSD (both port sdft.py’s advantage = teacher_logprob - student_logprob), following “Reinforcement Learning via Self-Distillation” (Hübotter et al., 2026): “the current model conditioned on feedback as a self-teacher… without any external teacher or explicit reward model.” That paper ships no reference implementation the way SDFT does in tinker_cookbook — the mechanics below are sdft.py’s, not a separate derivation. (Distinct from the unrelated, same-acronym “sDPO: Don’t Use Your Data All at Once” — sequential preference-data subsetting for offline DPO, not self-distillation.) Whitney mapping: Same as OPSD — teacher_loop.py’s frozen second run — with the teacher prompt built from feedback/critique instead of a golden answer. Fail closed: sdpo.py — feedback_prompt_tokens() is a placeholder for your own tokenized feedback-conditioned prompt.

Implementing with a coding agent

Point an agent at this page and GRPO. Ask it to:

Read capabilities for the target model.
Start from the matching file under examples/training/http/ — most entries already have a real build_loss; extend it rather than re-deriving the loop from prose.
Never import a provider SDK or substitute generic GRPO when the contract forbids it.

See Coding agents for MCP and skill setup.

Get started

How it works

Cookbooks

Coding agents

More algorithms

Shared Whitney loop

GRPO without standard deviation

Length-normalized GRPO

Dr.GRPO

PPO

CISPO

GSPO

DPPO

DAPO

SAPO

RLOO

REINFORCE++

MaxRL

On-policy distillation (OPD)

On-policy self-distillation (OPSD)

SDPO (Self-Distillation Policy Optimization)

Implementing with a coding agent

​Shared Whitney loop

​GRPO without standard deviation

​Length-normalized GRPO

​Dr.GRPO

​PPO

​CISPO

​GSPO

​DPPO

​DAPO

​SAPO

​RLOO

​REINFORCE++

​MaxRL

​On-policy distillation (OPD)

​On-policy self-distillation (OPSD)

​SDPO (Self-Distillation Policy Optimization)

​Implementing with a coding agent

Shared Whitney loop

GRPO without standard deviation

Length-normalized GRPO

Dr.GRPO

PPO

CISPO

GSPO

DPPO

DAPO

SAPO

RLOO

REINFORCE++

MaxRL

On-policy distillation (OPD)

On-policy self-distillation (OPSD)

SDPO (Self-Distillation Policy Optimization)

Implementing with a coding agent