> ## Documentation Index
> Fetch the complete documentation index at: https://staging.docs.trywhitney.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation

> Score model outputs with a caller-owned evaluation loop.

## Overview

Evaluation stays in your application. Whitney provides sampling primitives; you
hold prompts, references, rubrics, and scoring logic.

There is no hosted evaluator, dataset upload route, or managed eval job.

## Prerequisites

1. `GET /v1/training/capabilities` — confirm `sample` (and `logprobs` if needed).
2. Create a session and LoRA run, or reuse a run with a synced sampler.

## Evaluation loop

```text theme={null}
save_weights_for_sampler(version N)
  → sample(version N)
  → caller scores outputs locally
```

For each evaluation prompt:

1. Build a `SampleRequest` with `required_weight_version` set to the synced version.
2. Poll the sample operation to completion.
3. Run your scorer (exact match, rubric, model judge, execution check, etc.) in
   your own infrastructure.

`examples/training/http/eval_loop.py` is a complete, runnable implementation:
it samples once, then calls a `--score MODULE:CALLABLE` you supply.

```python score.py theme={null}
def score(sample: dict) -> dict:
    results = []
    for sequence in sample["sequences"]:
        # decode sequence["tokens"] and grade it against your reference
        results.append({"correct": ...})
    return {"accuracy": sum(r["correct"] for r in results) / len(results)}
```

```bash theme={null}
python examples/training/http/eval_loop.py \
  --provider modal --model Qwen/Qwen3.5-0.8B \
  --run-id <run_id> --sampler-id <sampler_id> \
  --sample-json sample.json --score score:score
```

Log results to your observability stack. Whitney does not host eval datasets or
Langfuse/Braintrust connectors in v1.

## When to sample vs export

* Use `sample` for live generations against a trained adapter.
* Use `export_lora` when you need a portable artifact for offline evaluation in
  another environment.

## Cleanup

Finish the run and close the session when evaluation completes, or `cancel` on
failure.