> ## Documentation Index
> Fetch the complete documentation index at: https://staging.docs.trywhitney.com/llms.txt
> Use this file to discover all available pages before exploring further.

# More algorithms

> Implement algorithm variants and distillation objectives as Whitney primitive loops.

Every algorithm on this page extends the [GRPO](/cookbooks/grpo) primitive
spine unless noted. Your code owns the math; Whitney owns ordered execution.
Every entry has a real, runnable file under
[`examples/training/http/`](https://github.com/try-whitney/whitney/tree/main/examples/training/http) —
either a `build_loss` callable plugged into `rl_loop.py` or `teacher_loop.py`,
or (for `sapo`, the one entry the provider itself doesn't expose) a
non-allocating capability-index stub that explains exactly what is missing.
See [Runnable reference implementations](/cookbooks/index#runnable-reference-implementations)
for the full table.

⚠️ **`loss_config` reaching the provider at all is a recent fix.** Whitney's
Modal trainer bridge (`training/runtime/modal_unified_trainer.py`) used to
call the provider's `forward_backward`/`forward` with only `(data,
loss_function)` — silently dropping every caller's `loss_config`, so PPO's
clip range, DPPO's divergence thresholds, and similar per-loss tuning always
ran on the provider's own defaults regardless of what a caller set. That's
now threaded through; confirm it with a live smoke (two otherwise-identical
requests with different `loss_config` values producing different results)
before trusting a non-default value in production.

## Shared Whitney loop

```text theme={null}
GET /v1/training/capabilities
  → POST /v1/training/sessions
  → POST /v1/training/sessions/{id}/runs
  → repeat per cycle:
      save_weights_for_sampler(N) → sample(N) → [optional logprobs]
      → build_loss(sample, cycle, request, ctx) -> ForwardRequest
      → forward_backward → optim_step
  → save_state / export_lora → finish → close
```

Required RL operations for most entries:
`forward_backward`, `optim_step`, `save_weights_for_sampler`, `sample`, `logprobs`.

Before allocating GPU work, confirm the capability row for your provider and
model advertises every operation you use. If the contract cannot be represented
in Whitney JSON primitives, stop before creating a session.

`GET /v1/training/capabilities` does not return a loss-function list — it
cannot, because Whitney forwards `loss_function` straight through to the
provider. `examples/training/http/whitney_datums.py`'s
`PROVIDER_LOSS_SUPPORT`/`require_advertised_loss` is a hand-maintained,
caller-side table of what each provider actually accepts
(`cross_entropy`/`importance_sampling`/`ppo`/`cispo` on both; `dro`
Tinker-only; `gspo`/`ppo_critic`/`dppo` Modal-only) — a hint that saves a
wasted allocation, not a live capability.

***

## GRPO without standard deviation

**Contract:** group advantage without standard-deviation normalization.

**Caller math:** After sampling, compute group-mean advantages only. Do not
divide by per-group standard deviation.

**Whitney mapping:** Same loop as [GRPO](/cookbooks/grpo). Encode the no-std
objective in `ForwardRequest.loss_function` and `loss_config`.

**Fail closed:** Do not use generic GRPO with std normalization.

**Code:** [`grpo_nostd.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/grpo_nostd.py) —
`group_mean_advantages(rewards, normalize_std=False)`, upstream
`tinker_cookbook`'s own default.

```python theme={null}
advantages = group_mean_advantages(rewards, normalize_std=False)
```

***

## Length-normalized GRPO

**Contract:** explicit token or sequence length-normalized loss.

**Caller math:** Apply your documented length weighting when building the loss
tensor inputs inside `ForwardRequest`.

**Whitney mapping:** Standard GRPO loop; length normalization lives entirely in
caller-constructed `loss_inputs` and `loss_config`.

**Fail closed:** Whitney does not apply length normalization implicitly.

**Code:** [`grpo_len_norm.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/grpo_len_norm.py) —
divide each trajectory's scalar advantage by its own completion length before
padding.

```python theme={null}
advantages = group_mean_advantages(rewards, normalize_std=True)
normalized = [length_normalize(a, len(seq["tokens"])) for a, seq in zip(advantages, sequences)]
```

***

## Dr.GRPO

**Contract:** no-std, length-normalized GRPO with no implicit KL penalty.

**Caller math:** Combine no-std group advantages, explicit length weighting, and
omit any implicit KL term your generic GRPO helper might add.

**Whitney mapping:** Standard GRPO loop with a custom `ForwardRequest`.

**Fail closed:** Not interchangeable with generic GRPO or length-normalized GRPO
alone.

**Code:** [`drgrpo.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/drgrpo.py) —
composes `grpo_nostd.py` and `grpo_len_norm.py`; "no implicit KL" needs no
extra code since neither `rl_loop.py` nor `whitney_datums.rl_datum` ever adds
one on their own.

```python theme={null}
advantages = group_mean_advantages(rewards, normalize_std=False)
normalized = [length_normalize(a, len(seq["tokens"])) for a, seq in zip(advantages, sequences)]
```

***

## PPO

**Contract:** policy plus first-class critic/value network and GAE lifecycle.

**Caller math:** Group-relative advantages, same as GRPO — no critic. Reading
the pinned Modal/SkyRL source settles this: `ppo_policy_loss` reads
`(log_probs, old_log_probs, advantages, config, loss_mask, rollout_logprobs)`,
the same four tensors as `importance_sampling`. The critic-needing variant is
the separately advertised `ppo_critic`, not plain `ppo`.

**Whitney mapping:** `ppo` is a real loss name on both providers. The clip
range is set two different ways per provider, same key names:

* **Tinker Cloud:** `clip_low_threshold`/`clip_high_threshold` as per-token
  tensors (`rl_datum(..., clip_low=, clip_high=)`).
* **Modal:** the same two names as **scalar** `loss_fn_config` entries
  (`skyrl/tinker/api.py`'s `_ALLOWED_KEYS_BY_LOSS_FN`) — see the
  `loss_config` note above.

**Fail closed:** [`ppo.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/ppo.py)
gates with `require_advertised_loss(ctx.provider, "ppo")` and never sends
`ppo_critic`.

```python theme={null}
tinker_clip = ctx.provider == "tinker"
data = [
    rl_datum(
        prompt_tokens, seq["tokens"], seq["logprobs"], adv,
        clip_low=PPO_CLIP_LOW if tinker_clip else None,
        clip_high=PPO_CLIP_HIGH if tinker_clip else None,
    )
    for seq, adv in zip(sequences, advantages)
]
loss_config = {} if tinker_clip else {"clip_low_threshold": PPO_CLIP_LOW, "clip_high_threshold": PPO_CLIP_HIGH}
return forward_backward_payload(data, "ppo", loss_config)
```

***

## CISPO

**Contract:** provider-advertised CISPO importance-sampling loss and data layout.

**Caller math:** Build standard GRPO-shaped data (group-mean advantages over
`rl_datum`); the only difference from generic GRPO is `loss_function`.

**Whitney mapping:** `cispo` is in both providers' advertised loss sets. On
Modal, `clip_low_threshold`/`clip_high_threshold` are valid scalar
`loss_config` keys (same mechanism as PPO, above); Tinker Cloud's CISPO
`loss_config` contract isn't published anywhere reachable from this repo, so
`cispo.py` leaves it empty there.

**Fail closed:** Never fall back to generic importance sampling or GRPO —
set `loss_function` explicitly.

**Code:** [`cispo.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/cispo.py) —
gated with `require_advertised_loss(ctx.provider, "cispo")` before building
the payload.

```python theme={null}
require_advertised_loss(ctx.provider, "cispo")
data = [rl_datum(prompt_tokens, seq["tokens"], seq["logprobs"], adv) for seq, adv in zip(sequences, advantages)]
loss_config = {"clip_low_threshold": CISPO_CLIP_LOW, "clip_high_threshold": CISPO_CLIP_HIGH} if ctx.provider == "modal" else {}
return forward_backward_payload(data, "cispo", loss_config)
```

***

## GSPO

**Contract:** provider-advertised GSPO loss and data contract.

**Caller math:** Same group-relative, no-critic shape as PPO/CISPO — GSPO's
"sequence-level importance ratios" are computed by the provider from the
same `weights`/`logprobs` tensors; nothing extra to build caller-side.

**Whitney mapping:** `gspo` is a real, Modal-only loss name — absent from
Tinker Cloud's `LossFnType` entirely. Its recommended `sequence_mean`
reduction mode is **not** a settable `loss_config` key on the pinned SkyRL
SHA (only `clip_low_threshold`/`clip_high_threshold` are), so it runs on
whatever Whitney's deployed default uses.

**Fail closed:** [`gspo.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/gspo.py)
gates to Modal only via `require_advertised_loss`. Not a generic
importance-sampling fallback.

***

## DPPO

**Contract:** DPPO plus critic/value lifecycle.

**Caller math:** No critic needed — `dppo_policy_loss` replaces PPO's ratio
clipping with a divergence-based binary mask computed from
`rollout_logprobs`, reading the same tensor set as every other RL loss here.

**Whitney mapping:** `dppo` is a real, Modal-only loss name. Its divergence
variant (`binary_tv` vs `binary_kl`) is fixed by Whitney's deployed default —
not a settable `loss_config` key; only the divergence thresholds
(`delta_low`/`delta_high`) are.

**Fail closed:** [`dppo.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/dppo.py)
gates to Modal only. Absent from Tinker Cloud's `LossFnType` entirely.

```python theme={null}
loss_config = {"delta_low": DPPO_DELTA_LOW, "delta_high": DPPO_DELTA_HIGH}
return forward_backward_payload(data, "dppo", loss_config)
```

***

## DAPO

**Contract:** dual clipping, dynamic rejection or resampling, overlong-sample
filtering.

**Caller math:** After sampling, filter overlong or invalid outputs, resample
per your declared policy, compute advantages on the retained group, apply upper
and lower clipping, then build `ForwardRequest`.

**Whitney mapping:** Multiple `sample` cycles may be needed per optimizer step.
`build_loss` receives a `LossContext` with a live client handle, so it can
issue additional `sample` operations itself, inside one cycle, rather than
needing a separate loop file. Each resample is still a real, billed, ordered
operation. Neither provider exposes a dedicated "dual-clip" loss name on the
wire — only `regular`/`ppo`-shaped clipping is selectable — so this
approximates DAPO with `loss_function="ppo"` plus a clip range, the same
per-provider mechanism as the PPO section above, not the paper's asymmetric
dual clip.

**Fail closed:** Filtering and resampling semantics must be caller-visible; do
not pretend one generic sample batch is DAPO.

**Code:** [`dapo.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/dapo.py):

```python theme={null}
def is_overlong(sequence):
    return sequence.get("stop_reason") == "length"

retained = [s for s in sequences if not is_overlong(s)]
while len(retained) < MIN_GROUP_SIZE:
    resampled = ctx.client.sampler_operation(ctx.run_id, ctx.sampler_id, "sample", request)
    sequences += resampled["sequences"]
    retained = [s for s in sequences if not is_overlong(s)]
tinker_clip = ctx.provider == "tinker"
data = [
    rl_datum(
        prompt_tokens, s["tokens"], s["logprobs"], adv,
        clip_low=DAPO_CLIP_LOW if tinker_clip else None,
        clip_high=DAPO_CLIP_HIGH if tinker_clip else None,
    )
    for s, adv in zip(retained, advantages)
]
loss_config = {} if tinker_clip else {"clip_low_threshold": DAPO_CLIP_LOW, "clip_high_threshold": DAPO_CLIP_HIGH}
return forward_backward_payload(data, "ppo", loss_config)
```

***

## SAPO

**Contract:** provider-native SAPO loss with explicit tau parameters.

**Caller math:** N/A — this is the one entry in the index that cannot be
sent today, not a caller-side gap.

**Whitney mapping:** The pinned SkyRL SHA does have a real, working
`sapo_policy_loss` internally — but Whitney's Modal adapter calls through
SkyRL's Tinker-*compatible* API server, whose request model hard-gates the
selectable loss to a fixed `Literal` of seven names. `"sapo"` isn't one of
them: a `loss_fn="sapo"` request is rejected by SkyRL's own validation
before it ever reaches the registry that supports it.

**Fail closed:** [`sapo.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/sapo.py)
stays a stub — not because the caller-side contract is unclear, but because
the provider genuinely does not expose this loss on the pinned SHA. Do not
substitute a generic clipping loss.

***

## RLOO

**Contract:** leave-one-out baseline plus reference or KL logprobs.

**Caller math:** Compute leave-one-out baselines across grouped samples. Fetch
reference logprobs via `logprobs` when KL is part of the objective.

**Whitney mapping:** `sample` → `logprobs` (reference) → `forward_backward` with
RLOO-encoded loss.

**Fail closed:** Group-mean GRPO is not a valid RLOO substitute.

**Code:** [`rloo.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/rloo.py) —
the leave-one-out baseline only; the optional reference-KL term is not
included (see REINFORCE++ for that pattern via `teacher_loop.py`).

```python theme={null}
advantages = leave_one_out_baseline(rewards)  # reward[i] - mean(rewards excluding i)
```

***

## REINFORCE++

**Contract:** REINFORCE++ with K2 and reference-KL semantics.

**Caller math:** Implement K2 baseline and reference-KL terms in caller code.
Use `logprobs` for reference policy probabilities.

**Whitney mapping:** A single run's sampler only ever serves its *current*
weights, so a genuine frozen reference needs a second Whitney run —
`teacher_loop.py` creates one, syncs it once, and never trains it, then
supplies its `logprobs` on the student's own generated tokens to `build_loss`
every cycle.

**Fail closed:** Requires advertised K2/reference-KL semantics.

**Code:** [`reinforce_pp.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/reinforce_pp.py) —
the K2 KL estimator, `0.5 * (student_logprob - reference_logprob) ** 2`, is one
reasonable reading of "K2 semantics", not a literature citation:

```python theme={null}
per_token_advantage = [
    group_advantage - KL_COEF * 0.5 * (student_lp - reference_lp) ** 2
    for student_lp, reference_lp in zip(student_logprobs, reference_logprobs)
]
```

***

## MaxRL

**Contract:** explicit MaxRL estimator.

**Caller math:** Encode the MaxRL objective directly in `ForwardRequest`. No
provider configuration name substitutes for this contract.

**Whitney mapping:** Standard RL operations; estimator logic is entirely caller-owned.

**Fail closed:** Do not map MaxRL to generic GRPO.

**Code:** [`maxrl.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/maxrl.py) —
no canonical MaxRL formula exists; this is one reasonable, clearly-editable
choice (a leave-one-out "beat the group's best" estimator), not a spec.

```python theme={null}
def max_relative_advantage(rewards):
    return [r - max(rewards[:i] + rewards[i + 1 :]) for i, r in enumerate(rewards)]
```

***

## On-policy distillation (OPD)

**Contract:** separate teacher or reference client and isolated teacher-data
exchange.

**Caller math:** Run teacher inference outside the training run (or via a
separate sampler) and pass teacher logits or targets into your distillation
loss.

**Whitney mapping:** `teacher_loop.py` creates a second Whitney run for the
teacher (any model — pass `--teacher-model` for a different, typically
larger, model), syncs its sampler once, and never trains or samples it: it
only answers `logprobs` on the student's own generated tokens. There is no
external reward — the signal is purely how much the teacher would have
preferred the student's choices.

**Fail closed:** Do not mix teacher and student weight versions on the same
sampler without explicit versioning.

**Code:** [`opd.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/opd.py) —
port of `tinker_cookbook/distillation/train_on_policy.py`'s
`incorporate_kl_penalty`: reverse KL as `log p - log q`, i.e.
`advantage = teacher_logprob - student_logprob`, per completion token.

```python theme={null}
per_token_advantage = [
    teacher_lp - student_lp for student_lp, teacher_lp in zip(student_logprobs, teacher_logprobs)
]
```

***

## On-policy self-distillation (OPSD)

**Contract:** teacher = the same base model, conditioned on a golden-answer
demonstration; student trains to match it over its own on-policy rollouts.

**Caller math:** Port of `tinker_cookbook/distillation/sdft.py` — Self-Distillation
Fine-Tuning (SDFT), ["Self-Distillation Enables Continual Learning"](https://arxiv.org/abs/2601.19897)
(Shenfeld et al., 2026) — the technique current survey literature groups
under "on-policy self-distillation". This ports SDFT's per-token
importance-sampling fallback (`Config.topk=0`):
`advantage = teacher_logprob - student_logprob`. SDFT's primary,
paper-validated mode (top-K soft-label `cross_entropy`, default `topk=20`)
needs a `(N, K)`-shaped soft-target datum from Whitney's
`topk_prompt_logprobs` sample field — a real extension, not implemented here.

**Whitney mapping:** `teacher_loop.py`'s second Whitney run, synced once and
never trained — SDFT's own default (`teacher_sync_every=None`, "works
comparably to EMA in our experiments"). The teacher prompt is `sdft.py`'s
own `DEFAULT_DEMO_TEMPLATE` (question + a golden answer as an in-context
demonstration), tokenized on the caller side — Whitney's HTTP contract
speaks only token IDs.

**Fail closed:** [`opsd.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/opsd.py) —
`golden_answer_prompt_tokens()` is a placeholder that raises until you supply
a real tokenized demonstration prompt.

```python theme={null}
teacher_prompt = golden_answer_prompt_tokens()  # tokenized DEFAULT_DEMO_TEMPLATE
for sequence in sequences:
    full_tokens = teacher_prompt + sequence["tokens"]
    result = ctx.client.sampler_operation(ctx.teacher_run_id, ctx.teacher_sampler_id, "logprobs", {"input": {"token_ids": full_tokens}})
    teacher_lps = result["logprobs"][-len(sequence["tokens"]):]
    advantage = [t - s for s, t in zip(sequence["logprobs"], teacher_lps)]
```

***

## SDPO (Self-Distillation Policy Optimization)

**Contract:** teacher = the same base model, conditioned on feedback about
the student's own prior attempt (not a full reference answer — that's OPSD).

**Caller math:** Same teacher-forced mechanism as OPSD (both port
`sdft.py`'s `advantage = teacher_logprob - student_logprob`), following
["Reinforcement Learning via Self-Distillation"](https://arxiv.org/abs/2601.20802)
(Hübotter et al., 2026): "the current model conditioned on feedback as a
self-teacher... without any external teacher or explicit reward model." That
paper ships no reference implementation the way SDFT does in
`tinker_cookbook` — the mechanics below are `sdft.py`'s, not a separate
derivation. (Distinct from the unrelated, same-acronym "sDPO: Don't Use Your
Data All at Once" — sequential preference-data subsetting for offline DPO,
not self-distillation.)

**Whitney mapping:** Same as OPSD — `teacher_loop.py`'s frozen second run —
with the teacher prompt built from feedback/critique instead of a golden
answer.

**Fail closed:** [`sdpo.py`](https://github.com/try-whitney/whitney/blob/main/examples/training/http/sdpo.py) —
`feedback_prompt_tokens()` is a placeholder for your own tokenized
feedback-conditioned prompt.

***

## Implementing with a coding agent

Point an agent at this page and [GRPO](/cookbooks/grpo). Ask it to:

1. Read capabilities for the target model.
2. Start from the matching file under `examples/training/http/` — most
   entries already have a real `build_loss`; extend it rather than
   re-deriving the loop from prose.
3. Never import a provider SDK or substitute generic GRPO when the contract
   forbids it.

See [Coding agents](/agents/coding-agents) for MCP and skill setup.