Hmm… maybe a process like below would make it easier to move forward:
My default would be not to make the sub-3B model itself contain all of the expertise. I would try to make the whole system behave like a product expert, while giving the small model a bounded role inside that system.
That leads me to roughly these answers:
- Can a small local model work? Possibly, for bounded tasks such as query rewriting, extracting missing information, summarizing retrieved evidence, producing structured drafts, and selecting from a small tool set. I would not assume that parameter count alone tells you whether it can handle the complete “experienced product expert” role.
- Is RAG enough? It is probably the default starting point for factual and versioned product knowledge, but not the whole solution. Troubleshooting, script validation, workflow construction, permissions, and execution need additional components.
- Would I start with CPT or SFT? No—not before identifying a failure that either one is likely to fix.
- Single model or multiple agents? I would start with one traceable workflow and one model. I would add agentic loops or specialized agents only where a fixed workflow demonstrably fails.
A reasonable first architecture might look like this:
user request
-> identify task, product version, environment, and missing inputs
-> retrieve from version-appropriate sources
-> optionally rerank and assemble evidence
-> small local model produces:
- an evidence-grounded answer,
- a clarification request,
- a structured workflow/script draft,
- or an abstention/escalation
-> deterministic validation where possible
-> return answer, evidence, warnings, and validation results
-> save a trace for evaluation
For troubleshooting, I would use a separate branch rather than treating it as ordinary document QA:
reported symptom
-> collect version/environment/configuration/logs/recent changes
-> maintain candidate causes and accumulated evidence
-> retrieve documentation and historical cases
-> run safe diagnostic checks
-> ask the next discriminating question
-> recommend a bounded next action or escalate
The important design choice is therefore not just “RAG vs. fine-tuning.” It is deciding which component is responsible for which kind of correctness.
| Responsibility |
Default owner |
| Finding current product facts |
Retriever plus version/authority filters |
| Explaining retrieved evidence |
Language model |
| Deciding that retrieval must occur |
Workflow policy for knowledge-dependent tasks |
| Checking parameter names and types |
Schema or API validator |
| Checking script syntax |
Parser/linter |
| Checking compatibility with a product version |
Versioned tests or metadata rules |
| Tracking a troubleshooting investigation |
Explicit diagnostic state |
| Approving a consequential action |
User or existing authorization process |
| Executing an action |
Gated tool, preferably with dry-run support |
| Confirming the action had the intended effect |
Test, observation, or state comparison |
| Deciding that the task is complete |
Task-specific completion criteria—not merely a fluent final message |
In other words, I would use the model mainly for the parts that are genuinely linguistic or ambiguous, and ordinary software for checks that can be made deterministically.
A practical default route
-
Choose one initial capability.
I would start with grounded documentation/configuration assistance, not autonomous troubleshooting and execution at the same time.
-
Create a small evaluation set before tuning anything.
Use real questions where possible, with expected evidence, applicable product version, and acceptable response behavior.
-
Inventory and normalize the sources.
Preserve version, module, platform, authority, publication state, deprecation status, headings, links, code blocks, and script provenance.
-
Run an oracle-context test on several candidate local models.
Give each model the correct evidence manually. This tests whether the generator can use the evidence before retrieval quality becomes a confounding variable.
-
Build a simple, inspectable RAG baseline.
Start with lexical plus dense retrieval, metadata filtering, and optionally a reranker. Avoid adding a graph or agent loop until a measured failure calls for it.
-
Add deterministic validators around configurations and scripts.
-
Implement troubleshooting as a stateful evidence-gathering workflow.
-
Use the remaining failures to decide whether SFT, CPT, or agentic search is justified.
A useful rule might be:
| Observed failure |
First place I would investigate |
| The required information is not in the corpus |
Content gap or expert capture process |
| The information exists but is not retrieved |
Chunking, metadata, query rewriting, hybrid retrieval, reranking |
| The right evidence is retrieved but incomplete |
Multi-document retrieval or context assembly |
| The complete evidence is supplied but the model misuses it |
Prompt/template, stronger model, or targeted SFT |
| The model cannot understand proprietary terminology or syntax even with good evidence |
Tokenization/domain representation, then possibly CPT |
| The model produces the right shape but wrong parameters |
Semantic tool-use evaluation and validators |
| A fixed workflow cannot decide what evidence/tool is needed next |
A bounded agentic loop |
| Generated scripts look plausible but fail |
Parser, schema, versioned tests, dry-run—not more prompting |
| The answer is correct but for the wrong product version |
Version-aware retrieval and explicit applicability checks |
The first experiment I would run is therefore not full RAG and not fine-tuning. It would be four small controls:
- model alone;
- model with manually supplied correct evidence;
- model with retrieved evidence;
- full workflow with tools and validators.
That separates model capacity, retrieval, orchestration, and tool integration failures.
A small evaluation set could include:
- direct documentation lookups;
- questions requiring two or more documents;
- configuration comparisons;
- version-sensitive questions;
- deprecated behavior;
- ambiguous questions where clarification is required;
- questions whose answer is absent;
- conflicting sources;
- workflow or script drafting;
- invalid parameters;
- troubleshooting over several turns;
- unsafe or unauthorized action requests.
This is broadly consistent with practical experience reported for enterprise software-documentation RAG: knowledge-base content and evaluation on real user questions can matter as much as changing the model. A particularly close technical example is RAG-EDA and its ORD-QA benchmark, with the accompanying RAG-EDA repository. It separates retriever, reranker, and generator adaptation rather than treating “the RAG system” as one indivisible component.
One safety boundary I would keep outside any folded details: do not move directly from generated advice to unrestricted script execution. Separate recommendation, draft generation, validation, approval, execution, effect verification, and completion. Local inference improves privacy characteristics, but it does not by itself prevent stale scripts, poisoned documents, excessive permissions, or destructive tool calls. The OWASP RAG Security Cheat Sheet is a useful checklist for the ingestion-to-tool-execution path.
Why I would decompose “product expert” into several capabilities
“Expert” sounds like one model property, but the requested behaviors have different inputs and different failure modes.
| Capability |
Required evidence/state |
Typical failure |
Better evaluation |
| Documentation QA |
Relevant current passages |
Wrong/missing retrieval |
Evidence recall and grounded answer |
| Concept explanation |
Several sources plus user level |
Correct but poorly scoped explanation |
Expert review for correctness and usefulness |
| Configuration guidance |
Version, environment, constraints |
Invented or incompatible values |
Schema/version validation |
| Workflow drafting |
Goals, available actions, examples |
Plausible but nonexistent actions |
Parse and execution tests |
| Troubleshooting |
Logs, state, history, competing causes |
Premature diagnosis |
Diagnostic trajectory and next-question quality |
| Best-practice recommendation |
Context, trade-offs, exceptions |
Generic advice stated as universal |
Applicability and exception checks |
| Execution |
Permissions and current system state |
Unintended side effects |
Dry-run, approval, state diff, rollback |
A system can be excellent at the first two and unsafe at the last two.
This also changes what “a small model can do” means. The useful question is not:
Can a 1B–3B model become an expert?
It is closer to:
Under a fixed context budget, tool contract, and latency target, which of these bounded functions can this particular model perform reliably enough?
For example, a small model might be adequate for:
- classifying the request into a small number of routes;
- extracting product version and environment fields;
- generating a search query;
- summarizing a few retrieved passages;
- filling a constrained JSON object;
- selecting one tool from a small, well-described list;
- producing a draft that is subsequently validated.
It may be inadequate for:
- open-ended multi-hop investigation over many heterogeneous sources;
- reliable diagnosis from incomplete evidence;
- integrating many long documents;
- making consequential decisions without validators;
- deciding on its own when all relevant evidence has been collected.
These are hypotheses to test, not conclusions that follow from the parameter count.
There is also a distinction between:
- being able to use a tool when explicitly told to;
- recognizing that a tool is required;
- choosing the correct tool;
- supplying semantically correct arguments;
- trusting the returned result over stale internal knowledge;
- stopping rather than repeatedly calling tools.
The current Transformers tool-use documentation is useful here. The model proposes a call, but the application executes the function and returns the result. The tool name, description, argument names, types, and chat template are all part of the contract. Tool use is therefore partly model capability and partly integration design.
I would test those behaviors independently:
A. No tool is available.
B. The tool is available but optional.
C. The workflow forces the tool call.
D. The correct tool result is injected directly.
E. The tool result conflicts with the model's likely prior knowledge.
This helps distinguish model ignorance, tool-selection failure, formatting/parser failure, and failure to use authoritative external evidence.
How I would prepare the knowledge sources
I would avoid flattening every source into interchangeable text chunks.
The sources establish different things:
| Source type |
Usually strongest for |
Common weakness |
| Official reference documentation |
Supported features and parameters |
May be terse or lag edge cases |
| Troubleshooting guides |
Known symptoms and standard fixes |
Often omit uncommon causes |
| Knowledge-base articles |
Operational procedures |
May be version-specific or duplicated |
| Tutorial videos |
UI sequence and demonstrations |
Transcripts lose visual context |
| Automation scripts |
Real API/action usage |
May contain legacy or environment-specific practices |
| Schemas and tests |
Machine-checkable contracts |
Do not explain intent |
| GitHub issues/support tickets |
Failures, exceptions, workarounds |
Variable authority and quality |
| Architecture documents |
System relationships and rationale |
Can become stale |
| Domain experts |
Tacit exceptions and escalation judgment |
Hard to scale and keep synchronized |
For each item or chunk, I would preserve metadata such as:
source_id
source_type
title
product/module
product_version_min
product_version_max
operating_system/environment
official/community/internal
draft/approved/deprecated
publication_or_commit_date
supersedes/superseded_by
script_language
required_permissions
source_url_or_repository_path
For versioned products, semantic similarity is not enough. An old page can be highly relevant while being operationally wrong. I would therefore make version applicability an explicit retrieval or validation condition, not merely another word in the prompt.
A recent preprint, VersionRAG, treats evolving technical documents as a distinct retrieval problem. Its specific architecture and reported numbers should not be assumed to transfer to another product, but the problem definition is useful: retrieval needs temporal/version validity, not only semantic relevance.
I would also preserve document structure:
- heading hierarchy;
- numbered procedures;
- warning/admonition blocks;
- tables;
- code blocks;
- links between reference, tutorial, troubleshooting, and migration pages;
- parent section information.
A short chunk can be good for retrieval while a larger parent section is better for answering. Those do not need to be the same unit.
A practical baseline could be:
metadata filtering
-> keyword/BM25 retrieval
-> dense retrieval
-> merge/deduplicate
-> optional reranking
-> expand selected chunks to parent sections
-> evidence sufficiency check
I would not add GraphRAG merely because the corpus is technical. A graph becomes more attractive if the actual questions repeatedly depend on explicit relationships such as:
- action A requires configuration B;
- feature C changed between versions;
- error D can arise from causes E/F/G;
- workflow step H produces artifact I consumed by step J.
Otherwise, preserving links and metadata may provide most of the value with less complexity.
For tutorial videos, I would initially index:
- transcript;
- chapter/title;
- timestamps;
- product version;
- referenced configuration/action names.
Only questions that depend on the visible UI, diagrams, or on-screen state need a heavier image/keyframe pipeline. That keeps the first local system tractable.
For scripts, I would store both searchable text and structured information if it can be extracted:
- imports/actions used;
- parameters;
- dependencies;
- input/output artifacts;
- target product version;
- permissions;
- known tests;
- owner/provenance;
- whether the script is current, illustrative, or production-approved.
That makes the scripts useful as examples without treating every historically used script as current best practice.
When I would use RAG, SFT, CPT, workflows, or agents
I would treat these as complementary mechanisms rather than competing architectures.
RAG
Use RAG for information that should remain inspectable and updateable:
- product facts;
- parameter definitions;
- current behavior;
- version-specific instructions;
- examples;
- troubleshooting evidence;
- links and citations.
A comparison study, Fine-Tuning or Retrieval?, found RAG more effective than the unsupervised fine-tuning setup it tested for injecting factual knowledge. That does not prove RAG always wins, especially for this product and these small models, but it supports using retrieval as the default baseline for changing facts rather than assuming CPT is required.
RAG is still not automatically reliable. The experience report Seven Failure Points When Engineering a Retrieval-Augmented Generation System is useful because it separates failures such as missing corpus content, failed retrieval, context consolidation, output specificity, and incomplete answers.
SFT
Use SFT when the problem is behavior rather than access to facts. Examples:
- asking for version/logs before diagnosing;
- following a required answer structure;
- citing evidence consistently;
- selecting tools;
- producing tool arguments;
- abstaining when evidence is insufficient;
- converting evidence into the platform’s workflow representation.
The training examples should resemble the actual deployed task. Raw manuals are not by themselves an SFT dataset.
If retrieval is generally correct but the model is distracted by irrelevant retrieved passages, RAFT is one relevant pattern: train in an open-book setting with relevant and distracting documents. I would view it as an optional post-baseline technique, not a prerequisite.
CPT / domain-adaptive pretraining
CPT may be worth a controlled experiment when the model struggles with the domain’s language or syntax even when the correct evidence is supplied. Possible signals include:
- proprietary tokens split very poorly;
- repeated misunderstanding of product-specific concepts;
- inability to model the script language;
- poor completion or representation of domain text independent of retrieval.
The foundational Don’t Stop Pretraining work supports domain-adaptive pretraining in the tasks it studied, but those were not this exact generative product-assistant setting. A few hundred pages are also not automatically enough to make CPT cost-effective or safe from overfitting/forgetting.
I would therefore compare:
base model
base model + RAG
base model + targeted SFT + RAG
base model + small CPT experiment + RAG
using the same held-out evaluation set.
Deterministic workflow
Use a workflow when the safe sequence is known:
extract version
-> retrieve only applicable sources
-> check evidence sufficiency
-> answer or request missing information
-> validate structured output
The current LangGraph workflow/agent guide makes a useful distinction: workflows follow predetermined paths, whereas agents dynamically decide their process and tool usage.
Agentic loop
Add an agentic loop when the system needs to decide, based on intermediate evidence:
- which source to search next;
- whether to inspect code, logs, or documentation;
- which diagnostic check best separates competing causes;
- whether the available evidence is contradictory;
- whether another retrieval pass is needed.
Keep it bounded:
maximum iterations
allowed tools per route
explicit state schema
evidence and tool-call trace
stop conditions
escalation condition
Multiple agents
I would use multiple agents only after identifying a genuine boundary such as:
- different permission sets;
- different context or tool sets;
- separately testable specialties;
- independently maintained components;
- a demonstrated quality/latency advantage over one workflow.
Complexity alone is not enough. The current LangChain multi-agent documentation also notes that many complex tasks can be handled by one agent with suitable tools and context.
Troubleshooting, scripts, validation, and human handoff
Troubleshooting is not merely “retrieve a troubleshooting page and answer.”
A useful state object might include:
{
"reported_symptoms": [],
"product_version": null,
"environment": {},
"configuration": {},
"logs_observed": [],
"recent_changes": [],
"candidate_causes": [
{
"cause": "",
"supporting_evidence": [],
"contradicting_evidence": [],
"status": "unconfirmed"
}
],
"missing_evidence": [],
"next_check": null,
"escalation_reason": null
}
The model can help propose and explain hypotheses, but the workflow should preserve the state and evidence between turns.
A recent preprint, DQA: Diagnostic Question Answering for IT Support, is close to this part of the problem. It explicitly maintains diagnostic state and aggregates evidence around candidate root causes. Its enterprise IT results do not establish performance for another software platform, but it provides a useful architecture to compare against ordinary conversational RAG.
The assistant should preferably ask the next question that distinguishes causes rather than asking for every possible detail at once. It should also be able to say:
- the evidence supports several causes;
- a required log or configuration value is missing;
- the available source is for another version;
- the proposed check is safe/read-only;
- this step requires escalation.
For scripts, I would distinguish these states:
retrieved example
-> generated draft
-> syntactically valid
-> schema/API-valid
-> compatible with target version
-> safe to dry-run
-> reviewed/approved
-> executed
-> observed effect verified
-> task completed
A fluent script should not be allowed to skip those transitions.
Where available, use:
- parser or compiler;
- formatter/linter;
- JSON/YAML/schema validation;
- API discovery or generated client definitions;
- product-version compatibility tests;
- static checks for secrets and dangerous actions;
- sandbox or read-only mode;
- dry-run;
- expected state diff;
- explicit approval;
- rollback or compensating action.
For local inference, llama.cpp’s server documentation is a useful implementation reference for GGUF serving, structured output, and tool-call support. Its grammar documentation also makes an important distinction: grammar/JSON-schema constraints can enforce an output form, but they do not prove that the contents are semantically correct.
For example:
{
"action": "set_parameter",
"parameter": "plausible_but_nonexistent_name",
"value": 42
}
may be perfectly schema-valid and still be wrong. The parameter must be checked against the product’s actual API/schema for the relevant version.
Human review also needs a usable handoff. “Ask a human” is not a complete fallback unless the reviewer receives:
- the original request;
- relevant product/environment state;
- sources and excerpts used;
- actions already attempted;
- validator results;
- unresolved hypotheses;
- permissions required;
- the reason escalation occurred.
The human also needs the authority and practical ability to intervene. Otherwise the system is merely transferring responsibility after losing the context needed to recover.
The same applies to correcting the knowledge base. A product expert does more than possess facts: they notice when a rule no longer applies, identify exceptions, stop unsafe procedures, and update guidance. It may be useful for the assistant to generate structured correction reports such as:
source considered stale or conflicting
observed behavior
affected product version/environment
evidence
temporary workaround
required reviewer
documents/tests/scripts that may need updating
That turns failures into maintenance input rather than merely asking the model to “self-correct.”
Evaluation and local deployment checks
I would keep retrieval and generation evaluation separate.
Corpus and retrieval
- Does an authoritative answer exist?
- Was the applicable version retrieved?
- Were all necessary documents retrieved?
- Were irrelevant or superseded sources included?
- Were tables, warnings, or code blocks preserved?
- Was the evidence sufficient to answer?
Generation
- Is every material claim supported?
- Is the answer complete?
- Does it distinguish fact from hypothesis?
- Does it state applicability and version?
- Does it ask for missing information when needed?
- Does it abstain when the corpus does not support an answer?
Tool and workflow behavior
- Was a tool needed?
- Was the correct tool selected?
- Were arguments syntactically and semantically valid?
- Was the returned result used faithfully?
- Did the workflow stop at the correct point?
- Were permission and approval gates respected?
Troubleshooting
- Did it preserve competing hypotheses?
- Did it avoid premature certainty?
- Did each question/check reduce uncertainty?
- Did it notice contradictory evidence?
- Did it escalate with a complete handoff?
Scripts and actions
- Did the output parse?
- Was it valid for the target version?
- Did it pass tests or dry-run?
- Did the resulting state match the expected diff?
- Could the action be reversed?
Local performance
Benchmark the complete pipeline, not only generation tokens per second:
- model load time;
- peak RAM;
- prompt-processing time;
- time to first token;
- generation speed;
- embedding time;
- reranking time;
- total latency;
- context length;
- number of retrieval/agent iterations;
- concurrency;
- repeated-run stability.
The exact quantization and runtime configuration should be part of the evaluation record.
A small model may generate quickly while the full system remains slow due to long contexts, reranking, repeated retrieval, or tool loops. Conversely, a slightly larger model may reduce retries enough to improve end-to-end latency.
I would also keep a frozen regression set and rerun it after changing:
- source documents;
- chunking;
- embedding model;
- reranker;
- generator;
- quantization;
- prompt;
- chat template;
- orchestration framework;
- tool schema;
- runtime version.
Store the trace:
input
resolved task/version/environment
rewritten searches
metadata filters
retrieved source IDs and scores
reranker output
final evidence supplied
model/runtime/template versions
tool calls and results
validator results
answer
human/evaluator feedback
This is often more valuable than a single overall score because it identifies the component that should be changed.
Nearby projects and references
These are not identical systems, but they may save some design work.
Technical software documentation
General implementation references
These repositories can be useful for inspecting ingestion, retrieval, citation, connector, UI, and deployment choices. Their feature lists are implementation references, not proof that a particular configuration will meet this system’s accuracy requirements.
Tuning and knowledge access
These are useful for distinguishing factual knowledge access, open-book behavior tuning, and domain-adaptive pretraining. None directly answers whether a particular sub-3B model will satisfy this product’s full requirements.
Failure analysis and difficult enterprise retrieval
The latter two use synthetic enterprise environments, so they are not substitutes for an evaluation set from the actual platform. They are useful sources of difficult categories: conflicting information, near-duplicates, multi-document evidence, constrained retrieval, and unanswerable questions.
Runtime, tools, and orchestration
Security
A staged roadmap
If I were starting from scratch, I would probably use these stages:
| Stage |
Scope |
Suggested exit condition |
| 0. Source and evaluation inventory |
Sources, versions, real questions, expected evidence |
Important source gaps and task categories are known |
| 1. Grounded QA |
Documentation/KB answers with evidence and abstention |
Acceptable retrieval and answer quality on held-out questions |
| 2. Configuration assistance |
Version-aware parameter lookup and comparison |
Outputs pass schema/version checks |
| 3. Workflow/script drafting |
Retrieve examples and generate non-executing drafts |
Drafts parse and pass deterministic validation |
| 4. Stateful troubleshooting |
Evidence collection and hypothesis tracking |
Useful next checks and safe escalation on replayed cases |
| 5. Controlled execution |
Read-only/dry-run, approval, effect verification |
Permissions, audit, rollback, and failure handling are tested |
| 6. Targeted optimization |
SFT/CPT/agent specialization based on measured gaps |
Improvement survives the frozen regression set and resource budget |
That sequence also provides a natural answer to the tuning question:
- If Stage 1 fails because retrieval is poor, tuning the generator is probably premature.
- If retrieval succeeds but the model cannot use correct evidence, compare a stronger model, prompt/template changes, and SFT.
- If domain syntax remains opaque under oracle context, run a small CPT experiment.
- If the fixed troubleshooting workflow cannot choose productive next checks, add a bounded agentic step.
- If several independently testable, permission-separated specialties emerge, then consider multiple agents.
So my main recommendation would be:
Build an inspectable, version-aware, retrieval-first system; allocate deterministic correctness to validators; keep troubleshooting state explicit; treat the small model as a bounded language/reasoning component; and let measured failure modes—not architecture fashion—decide when to add SFT, CPT, or agents.
That does not guarantee that a sub-3B model will meet the final target, but it should let you determine which role it can reliably perform, what needs to remain outside the model, and whether moving to a larger model would actually address the current bottleneck.