Qwen3.8-27B-ft-decision

Overview

This model is a fine-tuned version of Qwen3.8-27B on the typed-decisions dataset.

It takes a free-text input (a support ticket, an agent trace, an invoice, a security alert...) plus a schema of typed questions and returns, for every question, a calibrated probability distribution over its possible answers rather than a single label. Three question types are supported:

• choice — pick one option out of a set of named criteria (e.g. which team should handle a ticket).
• noul — binary yes/no decision (e.g. does the customer ask for a refund?).
• score — ordinal scale (e.g. urgency from "not urgent" to "today"); an expected score is also returned.

The dataset covers four workflows: agent_trace_observability, customer_service, invoice_processing and security_incidents (1.6k records in total, train/test splits). Gold labels are soft distributions aggregated from several annotators, so the model is trained to match the full distribution, not only the argmax.

A smaller sibling trained with the same recipe is available at mrm8488/Qwen3.5-0.8B-ft-decision-8k.

---

Training Configuration

• Dataset: LocalLLaMA/typed-decisions (all subset)
• Max Context Length: 4096
• Seed: 3407

Training Args

• Per Device Train Batch Size: 32
• Gradient Accumulation Steps: 1
  Effective batch size of 32.
• Warmup Steps: 10
• Total Training Epochs: 4
• Learning Rate: 2e-4
• LR Scheduler Type: cosine
• Weight Decay: 0.01
• Precision: bf16 (falls back to fp16 when bf16 is not supported)
• Evaluation / Save Strategy: epoch
• Load Best Model at End: True (selected by validation loss)

Training Progress

Epoch	Training Loss	Validation Loss
1	0.861723	0.837701
2	0.806615	0.836433
3	0.804017	0.830279
4	0.790902	0.829434

Validation loss decreases at every epoch, so the best checkpoint is also the final one (epoch 4).

LoRA Config

• rank: 16
• alpha: 16
• dropout: 0
• gradient checkpointing: "unsloth"
• random state: 3407

---

Evaluations

Metrics:

• accuracy: fraction of individual questions where the predicted label matches the gold argmax.
• record_accuracy: fraction of records where every question is answered correctly.
• ece: expected calibration error (lower is better).
• loss: cross-entropy against the soft gold distributions.

Base model (before fine-tuning)

Model	Accuracy	Record Accuracy	ECE ↓	Loss
Qwen3.8-27B (base)	43.25%	1.25%	0.1335	1.209

Test split

Model	Accuracy	Record Accuracy	ECE ↓	Loss
Ours (Qwen3.8-27B-ft-decision)	81.95%	42.0%	0.0361	1.308

Intended Use

The model is intended for research and experimentation on structured, calibrated decision-making: routing, triage, review queues and observability over workflows such as customer support, invoice processing, security incidents and agent traces. Because it outputs probabilities, it is well suited to thresholding (e.g. escalate to a human below a confidence level). Its performance should be validated on your own workflows and question schemas before deployment in production.

---

Example Usage (unsloth)

from unsloth import FastDecisionModel

model, tokenizer = FastDecisionModel.from_pretrained(
    model_name = "mrm8488/Qwen3.8-27B-ft-decision",
    max_seq_length = 4096,
)

FastDecisionModel.for_inference(model)

answers = FastDecisionModel.predict(
    model,
    tokenizer,
    "Hi, I was charged twice for invoice #4411. Please refund the duplicate today.",
    {
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this?",
            "criteria": {
                "billing": "invoices, payments, refunds",
                "technical": "bugs, outages, errors",
                "sales": "pricing, new plans",
            },
        },
        "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
        "urgency": {
            "type": "score",
            "instructions": "How urgent is this?",
            "criteria": ["not urgent", "soon", "today"],
        },
    },
)

for name, result in answers.items():
    print(name, result["answer"], {k: round(v, 3) for k, v in result["probabilities"].items()})

---

Limitations

• The training data is synthetic and English-only; expect lower accuracy on real-world, noisy or non-English inputs.
• Record-level accuracy (42% on test) is much lower than per-question accuracy: when a downstream action depends on several answers being right at once, gate it on the per-question probabilities.
• Gold labels reflect annotator disagreement; low-confidence predictions often correspond to genuinely ambiguous cases rather than model errors.
• Context length was limited to 4096 tokens during training; very long inputs (e.g. long agent traces) may need truncation.