value-steer safety value heads
Pre-trained scalar value heads for value-steer's
Value-Filtered Decoding (VFD) and dynamic-abstention runners. Each head scores the backbone's
final post-norm hidden state (the exact tensor lm_head consumes), per token, in fp32, and
predicts P(undesirable) — VFD keeps a sampled token when its value is below the threshold and
resamples when it is above.
Heads
Organized by backbone. Each <dataset>.bin is a bare ValueHead state dict (load with
value_steer.value_probe.load_value_head); its <dataset>.bin.meta.json sidecar carries the
feature contract and the calibrated threshold curve.
| Backbone | Datasets |
|---|---|
mistral/ — Mistral-7B-Instruct-v0.3 (hidden 4096) |
hh-rlhf, beavertails, pku_saferlhf |
llama/ — Llama-3.1-8B-Instruct (hidden 4096) |
hh-rlhf, beavertails, pku_saferlhf |
The head must match its backbone (hidden size + the exact decode-time feature); use each head with the backbone named in its sidecar.
Thresholds (conformal calibration curve)
Sidecars carry a curve, not a single number: the conformal posterior_threshold ĉ(α) at
α ∈ [0.05, 0.25, 0.45, 0.65, 0.85], where α is the false-intervention budget (the bound is
P(a safe trajectory is intervened on) ≤ α). Lower α → higher threshold → fewer interventions.
Pick the α that matches your risk tolerance and use that ĉ(α) as the VFD threshold; there is no
one "correct" operating point.
Calibration is decode-matched: thresholds were fit on the per-token values the VFD runner actually produces during decode (generate never-intervening, capture the decode hidden, judge the outputs with Llama-3.1-8B, then take the conformal quantile over the safe trajectories), on the same held-out prompts used by the reference implementation. Reported thresholds are single-stream; batched R>1 decode was measured and agrees to within a few thousandths.
Use
from value_steer.value_probe import load_value_head
head = load_value_head("mistral/hh-rlhf.bin", hidden_size=4096) # picks CUDA if available
Serve with vLLM (VFD; pick a threshold from the sidecar curve):
vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
--worker-cls value_steer.worker.ValueSteerWorker \
--additional-config '{"vfd": {"value_head_path": "mistral/hh-rlhf.bin", "threshold": 0.36, "num_candidates": 8}}'
See the value-steer repo for the runners, the feature contract, and training/calibration details.