GRPO

Overview

GRPO is the foundation for most reinforcement-learning objectives on Whitney. Your code owns prompts, rewards, grouping, advantages, and the loss payload. Whitney owns sampling, weight sync, and ordered primitive execution. Other algorithms in More algorithms extend this loop with different advantage estimators, clipping, or distillation semantics.

Prerequisites

GET /v1/training/capabilities — confirm forward_backward, optim_step, save_weights_for_sampler, sample, and optionally logprobs.
Create a session and LoRA run (see Quickstart).

Training loop

For each cycle:

save_weights_for_sampler(version N)
  → sample(version N)
  → caller computes rewards and group advantages
  → forward_backward(caller loss)
  → optim_step(explicit AdamW)

Run at least two full cycles so update, sync, and sample semantics are explicit.

Caller-owned math

After sample returns:

Score each completion with your reward function.
Group completions by prompt and compute advantages (for standard GRPO: mean-center within each group; optionally divide by standard deviation).
Build a forward JSON payload whose loss_function and loss_config encode your policy-gradient objective for those advantages.

examples/training/http/rl_loop.py is a complete, runnable implementation of this loop. It calls a --build-loss MODULE:CALLABLE you supply on every cycle — build_loss(sample, cycle, request, ctx) -> dict — so the only code you write is the reward function and the few lines below:

# source: examples/training/http/grpo_build_loss.py, using
# examples/training/http/whitney_datums.py (a stdlib port of
# tinker_cookbook's rl_loop.py group-mean advantage + datum construction)
from whitney_datums import all_zero, forward_backward_payload, group_mean_advantages, rl_datum


def reward(tokens: list[int]) -> float:
    ...  # your scoring function — decode tokens, grade the answer, return a scalar


def build_loss(sample, cycle, request, ctx):
    prompt_tokens = request["prompt"]["token_ids"]
    sequences = sample["sequences"]
    rewards = [reward(sequence["tokens"]) for sequence in sequences]
    advantages = group_mean_advantages(rewards, normalize_std=True)
    if all_zero(advantages):
        raise ValueError(f"cycle {cycle}: reward is uniform across the group, no gradient signal")
    data = [
        rl_datum(prompt_tokens, sequence["tokens"], sequence["logprobs"], advantage)
        for sequence, advantage in zip(sequences, advantages)
    ]
    return forward_backward_payload(data, "importance_sampling")

python examples/training/http/rl_loop.py \
  --provider modal --model Qwen/Qwen3.5-0.8B \
  --sample-json sample.json --build-loss grpo_build_loss:build_loss

Never substitute generic GRPO when your algorithm requires a different estimator. See More algorithms for variants — most of them are one small change to build_loss over the same loop.

Log probabilities

Use logprobs when your objective needs policy or reference log probabilities. Do not infer them from generated text. Sample, logprobs, and forward bodies use the typed JSON primitive contract.

Checkpoints

save_state → export_lora → finish → close session

Optionally call save_weights_for_sampler before each sample cycle.

Cleanup

On failure: cancel the run, poll until terminal, then close the session.

If the algorithm-specific loss cannot be represented in the current Whitney JSON and operation contract, stop before creating a session. Never silently substitute generic GRPO.

Get started

How it works

Cookbooks

Coding agents

Overview

Prerequisites

Training loop

Caller-owned math

Log probabilities

Checkpoints

Cleanup

​Overview

​Prerequisites

​Training loop

​Caller-owned math

​Log probabilities

​Checkpoints

​Cleanup

Overview

Prerequisites

Training loop

Caller-owned math

Log probabilities

Checkpoints

Cleanup