Evaluation

Overview

Evaluation stays in your application. Whitney provides sampling primitives; you hold prompts, references, rubrics, and scoring logic. There is no hosted evaluator, dataset upload route, or managed eval job.

Prerequisites

GET /v1/training/capabilities — confirm sample (and logprobs if needed).
Create a session and LoRA run, or reuse a run with a synced sampler.

Evaluation loop

save_weights_for_sampler(version N)
  → sample(version N)
  → caller scores outputs locally

For each evaluation prompt:

Build a SampleRequest with required_weight_version set to the synced version.
Poll the sample operation to completion.
Run your scorer (exact match, rubric, model judge, execution check, etc.) in your own infrastructure.

examples/training/http/eval_loop.py is a complete, runnable implementation: it samples once, then calls a --score MODULE:CALLABLE you supply.

score.py

def score(sample: dict) -> dict:
    results = []
    for sequence in sample["sequences"]:
        # decode sequence["tokens"] and grade it against your reference
        results.append({"correct": ...})
    return {"accuracy": sum(r["correct"] for r in results) / len(results)}

python examples/training/http/eval_loop.py \
  --provider modal --model Qwen/Qwen3.5-0.8B \
  --run-id <run_id> --sampler-id <sampler_id> \
  --sample-json sample.json --score score:score

Log results to your observability stack. Whitney does not host eval datasets or Langfuse/Braintrust connectors in v1.

When to sample vs export

Use sample for live generations against a trained adapter.
Use export_lora when you need a portable artifact for offline evaluation in another environment.

Cleanup

Finish the run and close the session when evaluation completes, or cancel on failure.

Get started

How it works

Cookbooks

Coding agents

Overview

Prerequisites

Evaluation loop

When to sample vs export

Cleanup

​Overview

​Prerequisites

​Evaluation loop

​When to sample vs export

​Cleanup

Overview

Prerequisites

Evaluation loop

When to sample vs export

Cleanup