> ## Documentation Index
> Fetch the complete documentation index at: https://staging.docs.trywhitney.com/llms.txt
> Use this file to discover all available pages before exploring further.

# GRPO

> Run a caller-owned group-relative policy optimization loop.

## Overview

GRPO is the foundation for most reinforcement-learning objectives on Whitney.
Your code owns prompts, rewards, grouping, advantages, and the loss payload.
Whitney owns sampling, weight sync, and ordered primitive execution.

Other algorithms in [More algorithms](/cookbooks/algorithms) extend this loop
with different advantage estimators, clipping, or distillation semantics.

## Prerequisites

1. `GET /v1/training/capabilities` — confirm `forward_backward`, `optim_step`,
   `save_weights_for_sampler`, `sample`, and optionally `logprobs`.
2. Create a session and LoRA run (see [Quickstart](/quickstart)).

## Training loop

For each cycle:

```text theme={null}
save_weights_for_sampler(version N)
  → sample(version N)
  → caller computes rewards and group advantages
  → forward_backward(caller loss)
  → optim_step(explicit AdamW)
```

Run at least two full cycles so update, sync, and sample semantics are explicit.

### Caller-owned math

After `sample` returns:

1. Score each completion with your reward function.
2. Group completions by prompt and compute advantages (for standard GRPO: mean-center
   within each group; optionally divide by standard deviation).
3. Build a forward JSON payload whose `loss_function` and `loss_config` encode your
   policy-gradient objective for those advantages.

`examples/training/http/rl_loop.py` is a complete, runnable implementation of
this loop. It calls a `--build-loss MODULE:CALLABLE` you supply on every
cycle — `build_loss(sample, cycle, request, ctx) -> dict` — so the only code
you write is the reward function and the few lines below:

<CodeGroup>
  ```python grpo_build_loss.py theme={null}
  # source: examples/training/http/grpo_build_loss.py, using
  # examples/training/http/whitney_datums.py (a stdlib port of
  # tinker_cookbook's rl_loop.py group-mean advantage + datum construction)
  from whitney_datums import all_zero, forward_backward_payload, group_mean_advantages, rl_datum


  def reward(tokens: list[int]) -> float:
      ...  # your scoring function — decode tokens, grade the answer, return a scalar


  def build_loss(sample, cycle, request, ctx):
      prompt_tokens = request["prompt"]["token_ids"]
      sequences = sample["sequences"]
      rewards = [reward(sequence["tokens"]) for sequence in sequences]
      advantages = group_mean_advantages(rewards, normalize_std=True)
      if all_zero(advantages):
          raise ValueError(f"cycle {cycle}: reward is uniform across the group, no gradient signal")
      data = [
          rl_datum(prompt_tokens, sequence["tokens"], sequence["logprobs"], advantage)
          for sequence, advantage in zip(sequences, advantages)
      ]
      return forward_backward_payload(data, "importance_sampling")
  ```

  ```bash run.sh theme={null}
  python examples/training/http/rl_loop.py \
    --provider modal --model Qwen/Qwen3.5-0.8B \
    --sample-json sample.json --build-loss grpo_build_loss:build_loss
  ```
</CodeGroup>

Never substitute generic GRPO when your algorithm requires a different estimator.
See [More algorithms](/cookbooks/algorithms) for variants — most of them are
one small change to `build_loss` over the same loop.

### Log probabilities

Use `logprobs` when your objective needs policy or reference log probabilities.
Do not infer them from generated text.

Sample, logprobs, and forward bodies use the typed
[JSON primitive contract](/concepts/json-primitives).

## Checkpoints

```text theme={null}
save_state → export_lora → finish → close session
```

Optionally call `save_weights_for_sampler` before each sample cycle.

## Cleanup

On failure: `cancel` the run, poll until terminal, then `close` the session.

<Warning>
  If the algorithm-specific loss cannot be represented in the current Whitney JSON and operation contract, stop before
  creating a session. Never silently substitute generic GRPO.
</Warning>