More algorithms

Every algorithm on this page extends the GRPO primitive spine unless noted. Your code owns the math; Whitney owns ordered execution. Each section builds on the shared loop building blocks — datum construction, advantage helpers, and the build_loss contract. Distillation objectives (OPD, OPSD, SDPO) have their own group — see Distillation.

Shared Whitney loop

GET /v1/training/capabilities
  → POST /v1/training/sessions
  → POST /v1/training/sessions/{id}/runs
  → repeat per cycle:
      save_weights_for_sampler(N) → sample(N) → [optional logprobs]
      → build_loss(sample, cycle, request, ctx) -> ForwardRequest
      → forward_backward → optim_step
  → save_state / export_lora → finish → close

Required RL operations for most entries: forward_backward, optim_step, save_weights_for_sampler, sample, logprobs. Before allocating GPU work, confirm the capability row for your provider and model advertises every operation you use. If the contract cannot be represented in Whitney JSON primitives, stop before creating a session. GET /v1/training/capabilities does not return a loss-function list — it cannot, because Whitney forwards loss_function straight through to the provider. See Provider loss support for the caller-side table of what each provider actually accepts.

GRPO without standard deviation

Contract: group advantage without standard-deviation normalization. Caller math: After sampling, compute group-mean advantages only. Do not divide by per-group standard deviation. Whitney mapping: Same loop as GRPO. Encode the no-std objective in ForwardRequest.loss_function and loss_config. Fail closed: Do not use generic GRPO with std normalization.

advantages = group_mean_advantages(rewards, normalize_std=False)

Length-normalized GRPO

Contract: explicit token or sequence length-normalized loss. Caller math: Apply your documented length weighting when building the loss tensor inputs inside ForwardRequest. Whitney mapping: Standard GRPO loop; length normalization lives entirely in caller-constructed loss_inputs and loss_config. Fail closed: Whitney does not apply length normalization implicitly.

advantages = group_mean_advantages(rewards, normalize_std=True)
normalized = [length_normalize(a, len(seq["tokens"])) for a, seq in zip(advantages, sequences)]

Dr.GRPO

Contract: no-std, length-normalized GRPO with no implicit KL penalty. Caller math: Combine no-std group advantages, explicit length weighting, and omit any implicit KL term your generic GRPO helper might add. Whitney mapping: Standard GRPO loop with a custom ForwardRequest. “No implicit KL” needs no extra code — Whitney never adds one on its own. Fail closed: Not interchangeable with generic GRPO or length-normalized GRPO alone.

advantages = group_mean_advantages(rewards, normalize_std=False)
normalized = [length_normalize(a, len(seq["tokens"])) for a, seq in zip(advantages, sequences)]

PPO

Contract: policy plus first-class critic/value network and GAE lifecycle. Caller math: Group-relative advantages, same as GRPO — no critic. ppo’s policy loss reads (log_probs, old_log_probs, advantages, config, loss_mask, rollout_logprobs), the same four tensors as importance_sampling. The critic-needing variant is the separately advertised ppo_critic, not plain ppo. Whitney mapping: ppo is a real loss name on both providers. The clip range is set two different ways per provider, same key names:

Tinker Cloud: clip_low_threshold/clip_high_threshold as per-token tensors (rl_datum(..., clip_low=, clip_high=)).
Modal: the same two names as scalar loss_config entries — see Provider loss support.

Fail closed: Gate with require_advertised_loss(provider, "ppo") and never send ppo_critic.

tinker_clip = provider == "tinker"
data = [
    rl_datum(
        prompt_tokens, seq["tokens"], seq["logprobs"], adv,
        clip_low=PPO_CLIP_LOW if tinker_clip else None,
        clip_high=PPO_CLIP_HIGH if tinker_clip else None,
    )
    for seq, adv in zip(sequences, advantages)
]
loss_config = {} if tinker_clip else {"clip_low_threshold": PPO_CLIP_LOW, "clip_high_threshold": PPO_CLIP_HIGH}
return forward_backward_payload(data, "ppo", loss_config)

CISPO

Contract: provider-advertised CISPO importance-sampling loss and data layout. Caller math: Build standard GRPO-shaped data (group-mean advantages over rl_datum); the only difference from generic GRPO is loss_function. Whitney mapping: cispo is in both providers’ advertised loss sets. On Modal, clip_low_threshold/clip_high_threshold are valid scalar loss_config keys (same mechanism as PPO, above); Tinker Cloud’s CISPO loss_config contract isn’t published, so leave it empty there. Fail closed: Never fall back to generic importance sampling or GRPO — set loss_function explicitly.

require_advertised_loss(provider, "cispo")
data = [rl_datum(prompt_tokens, seq["tokens"], seq["logprobs"], adv) for seq, adv in zip(sequences, advantages)]
loss_config = {"clip_low_threshold": CISPO_CLIP_LOW, "clip_high_threshold": CISPO_CLIP_HIGH} if provider == "modal" else {}
return forward_backward_payload(data, "cispo", loss_config)

GSPO

Contract: provider-advertised GSPO loss and data contract. Caller math: Same group-relative, no-critic shape as PPO/CISPO — GSPO’s “sequence-level importance ratios” are computed by the provider from the same weights/logprobs tensors; nothing extra to build caller-side. Whitney mapping: gspo is a real, Modal-only loss name — absent from Tinker Cloud entirely. Its recommended sequence_mean reduction mode is not a settable loss_config key (only clip_low_threshold/ clip_high_threshold are), so it runs on whatever Whitney’s deployed default uses. Fail closed: Gate to Modal only via require_advertised_loss. Not a generic importance-sampling fallback.

require_advertised_loss(provider, "gspo")
data = [rl_datum(prompt_tokens, seq["tokens"], seq["logprobs"], adv) for seq, adv in zip(sequences, advantages)]
loss_config = {"clip_low_threshold": GSPO_CLIP_LOW, "clip_high_threshold": GSPO_CLIP_HIGH}
return forward_backward_payload(data, "gspo", loss_config)

DPPO

Contract: DPPO plus critic/value lifecycle. Caller math: No critic needed — DPPO’s policy loss replaces PPO’s ratio clipping with a divergence-based binary mask computed from rollout_logprobs, reading the same tensor set as every other RL loss here. Whitney mapping: dppo is a real, Modal-only loss name. Its divergence variant (binary_tv vs binary_kl) is fixed by Whitney’s deployed default — not a settable loss_config key; only the divergence thresholds (delta_low/delta_high) are. Fail closed: Gate to Modal only. Absent from Tinker Cloud entirely.

loss_config = {"delta_low": DPPO_DELTA_LOW, "delta_high": DPPO_DELTA_HIGH}
return forward_backward_payload(data, "dppo", loss_config)

DAPO

Contract: dual clipping, dynamic rejection or resampling, overlong-sample filtering. Caller math: After sampling, filter overlong or invalid outputs, resample per your declared policy, compute advantages on the retained group, apply upper and lower clipping, then build ForwardRequest. Whitney mapping: Multiple sample cycles may be needed per optimizer step. build_loss receives a LossContext with a live client handle, so it can issue additional sample operations itself, inside one cycle, rather than needing a separate loop file. Each resample is still a real, billed, ordered operation. Neither provider exposes a dedicated “dual-clip” loss name on the wire — only regular/ppo-shaped clipping is selectable — so this approximates DAPO with loss_function="ppo" plus a clip range, the same per-provider mechanism as the PPO section above, not the paper’s asymmetric dual clip. Fail closed: Filtering and resampling semantics must be caller-visible; do not pretend one generic sample batch is DAPO.

def is_overlong(sequence):
    return sequence.get("stop_reason") == "length"

retained = [s for s in sequences if not is_overlong(s)]
while len(retained) < MIN_GROUP_SIZE:
    resampled = ctx.client.sampler_operation(ctx.run_id, ctx.sampler_id, "sample", request)
    sequences += resampled["sequences"]
    retained = [s for s in sequences if not is_overlong(s)]
tinker_clip = ctx.provider == "tinker"
data = [
    rl_datum(
        prompt_tokens, s["tokens"], s["logprobs"], adv,
        clip_low=DAPO_CLIP_LOW if tinker_clip else None,
        clip_high=DAPO_CLIP_HIGH if tinker_clip else None,
    )
    for s, adv in zip(retained, advantages)
]
loss_config = {} if tinker_clip else {"clip_low_threshold": DAPO_CLIP_LOW, "clip_high_threshold": DAPO_CLIP_HIGH}
return forward_backward_payload(data, "ppo", loss_config)

SAPO

Contract: provider-native SAPO loss with explicit tau parameters. Caller math: N/A — this is the one entry in the index that cannot be sent today, not a caller-side gap. Whitney mapping: Not selectable. The provider’s own Tinker-compatible API restricts the selectable loss to a fixed set that excludes SAPO — a loss_function="sapo" request is rejected by that validation before it reaches any code that might support it internally. Fail closed: Do not substitute a generic clipping loss for SAPO. Check Provider loss support — or call GET /v1/training/capabilities — before allocating.

RLOO

Contract: leave-one-out baseline plus reference or KL logprobs. Caller math: Compute leave-one-out baselines across grouped samples. Fetch reference logprobs via logprobs when KL is part of the objective. Whitney mapping: sample → logprobs (reference) → forward_backward with RLOO-encoded loss. Fail closed: Group-mean GRPO is not a valid RLOO substitute. The leave-one-out baseline only; an optional reference-KL term follows the same pattern as REINFORCE++ below via a frozen second run.

advantages = leave_one_out_baseline(rewards)  # reward[i] - mean(rewards excluding i)

REINFORCE++

Contract: REINFORCE++ with K2 and reference-KL semantics. Caller math: Implement K2 baseline and reference-KL terms in caller code. Use logprobs for reference policy probabilities. Whitney mapping: A single run’s sampler only ever serves its current weights, so a genuine frozen reference needs a second Whitney run, synced once and never trained, that supplies its logprobs on the student’s own generated tokens to build_loss every cycle. Fail closed: Requires advertised K2/reference-KL semantics. The K2 KL estimator, 0.5 * (student_logprob - reference_logprob) ** 2, is one reasonable reading of “K2 semantics”, not a literature citation:

per_token_advantage = [
    group_advantage - KL_COEF * 0.5 * (student_lp - reference_lp) ** 2
    for student_lp, reference_lp in zip(student_logprobs, reference_logprobs)
]

MaxRL

Contract: explicit MaxRL estimator. Caller math: Encode the MaxRL objective directly in ForwardRequest. No provider configuration name substitutes for this contract. Whitney mapping: Standard RL operations; estimator logic is entirely caller-owned. Fail closed: Do not map MaxRL to generic GRPO. No canonical MaxRL formula exists; this is one reasonable, clearly-editable choice (a leave-one-out “beat the group’s best” estimator), not a spec:

def max_relative_advantage(rewards):
    return [r - max(rewards[:i] + rewards[i + 1 :]) for i, r in enumerate(rewards)]

Implementing with a coding agent

Point an agent at this page and GRPO. Ask it to:

Read capabilities for the target model.
Start from the matching section here (or Distillation for OPD/OPSD/SDPO) and the shared loop building blocks — most entries already have a real build_loss snippet to extend rather than re-deriving the loop from prose.
Never import a provider SDK or substitute generic GRPO when the contract forbids it.

See Coding agents for MCP and skill setup.

Get started

How it works

Cookbooks

Distillation

Coding agents

More algorithms

Shared Whitney loop

GRPO without standard deviation

Length-normalized GRPO

Dr.GRPO

PPO

CISPO

GSPO

DPPO

DAPO

SAPO

RLOO

REINFORCE++

MaxRL

Implementing with a coding agent

​Shared Whitney loop

​GRPO without standard deviation

​Length-normalized GRPO

​Dr.GRPO

​PPO

​CISPO

​GSPO

​DPPO

​DAPO

​SAPO

​RLOO

​REINFORCE++

​MaxRL

​Implementing with a coding agent

Shared Whitney loop

GRPO without standard deviation

Length-normalized GRPO

Dr.GRPO

PPO

CISPO

GSPO

DPPO

DAPO

SAPO

RLOO

REINFORCE++

MaxRL

Implementing with a coding agent