Local LLM on M5 Pro, open webUI issue (cont’d)

[alaskaav8r]

This topic has background found here that was inadvertently marked closed New to This LLM

First off thanks @John6666 I have my testbench doc and files 90% complete. So I am almost ready to start testing in a more coherent way. However, I remembered an issue I ran into while setting my system up and went through my notes to check if it had been corrected.

Also to note, i am new to all this, not an IT worker, consider myself at the level slightly above a novice. I have to use Claude to explain things, help write up the technical stuff and help set up my stack. While working on my testing bench I remembered during setup Claude and I discovered that “native” mode does not work for RAG search.

Here are my specs:

MacBook M5 Max (18-core CPU, 40-core GPU, 128GB unified memory), macOS Tahoe

**•**	Open WebUI v0.10.2, self-hosted via Docker Compose (“myLLM”)

**•**	Docker Model Runner (llama.cpp-metal backend), not Ollama

**•**	Custom model: Gemma 4 (base ai/gemma4, 7.52B, Q4_K_M)

**•**	Embedding: SentenceTransformers (all-MiniLM-L6-v2)

**•**	Knowledge Base: mix of PDFs and hand-written markdown reference files

The issue is that Native mode’s query_knowledge_files and kb_exec both fail to retrieve from attached knowledge bases with ‘str’ object has no attribute ‘items’; Legacy mode works fine with the same model and knowledge base. I did open a bug report via github (#26880). Yesterday I updated open webUI and docker to their latest versions and the issue still persists.

For my stack I considered LM Studio but went with Docker for the containerization. I actually started with Ollama, running inside Docker. It hit a hard wall on my Mac, Docker Desktop couldn’t pass my M5 Max’s GPU through to Ollama’s container, so it ran CPU-only, plus a 28GB memory ceiling regardless of my 128GB. Docker Model Runner replaced it because it runs natively on the Mac host, full Metal GPU access, full RAM, no container ceiling.

Now I am running my models in “legacy” mode which is now unsupported so until (or if) they fix this issue I seem to be stuck with this limit.

I have a few thoughts and questions.

In my mind I think I should rethink using open webUI and consider using AnythingLLM as my frontend interface before starting serious testing unless anyone has any ideas how to solve my current issue. My thought is that if I need to switch frontends, why not do it now.

Would love to get your perspective on 1) how to fix my native mode issue or 2) using AnythingLLM.

Tim

[John6666]

For now, I ran a few lightweight checks on my side too. A few things became clearer, while a few others still depend on the exact details of your setup:

---

My direct answer would be:

1. I would not abandon Open WebUI yet.
2. I would continue building the test bench now instead of waiting for Native mode to be fixed.
3. I would keep Legacy as a temporary RAG control, not as the long-term target.
4. I would try AnythingLLM only as a small parallel A/B comparison, not as a one-way migration yet.

The reason is that your Legacy result already tells us something useful: the documents are usable, and at least one retrieval-and-answer path works with this model and Knowledge Base. But Native mode adds several extra moving parts, so Legacy working does not mean Native should automatically work.

Open WebUI’s current documentation makes that distinction fairly explicit: in Native mode, attached Knowledge is not automatically inserted into the prompt; the model must call tools such as list_knowledge and query_knowledge_files. Disabling Native restores the older automatic RAG injection path, while Full Context bypasses retrieval and injects the document directly. See Open WebUI Knowledge and Tool Calling Modes.

So I would treat your situation as:

A Native tool/integration problem, not yet evidence that your documents, embeddings, Gemma 4, Docker Model Runner, or Open WebUI are individually “bad.”

Your GitHub issue is already a solid report: it includes the environment, two Open WebUI versions, the Legacy control, the Native failure, kb_exec, the repeated calls, and the lack of a useful traceback. I would leave that issue as-is until there is one new piece of direct evidence, rather than adding more speculative links.

The practical route I would use

I would split the test bench into three independent tracks:

Track	Settings	What it measures
A. Model baseline	Knowledge OFF, tools OFF, web search OFF	The model/artifact/runtime itself
B. RAG control	Legacy, or Full Context for a very short document; web search OFF	Whether the system can answer from your documents
C. Native integration	Native tools ON, web search OFF, minimal Knowledge	Tool selection, argument transport, retrieval execution, result handling, loop stopping

For ordinary daily use, web search can still be ON when you want current information. For model, RAG, source-discipline, and tool tests, keep it OFF so a correct-looking web result cannot hide a broken Knowledge path.

Also, a normal prompt such as What is 7 plus 3? is not a function-calling test unless a real tool schema is present and the runtime records an actual tool call. A useful Native test needs a real tool and observable arguments/results.

Default path

If your priority is to start serious testing now:

• Continue with Open WebUI.
• Use Legacy as the temporary RAG control.
• Keep Native as a separate pass/fail integration test.
• Do not rebuild the Knowledge Base or change the embedding model first.
• Record the stack version for each test run.

If your priority is to have supported Native Knowledge working immediately:

• Run AnythingLLM beside Open WebUI.
• Keep the same DMR endpoint, model artifact, and tiny test document.
• Treat success there as a practical workaround, not proof that Open WebUI caused the original problem.

If your priority is to locate the failing layer:

• The highest-value evidence is the first failed tool call before the 49-call loop grows.
• A direct non-streaming DMR response is the cleanest version of that evidence.

What I tested, and what it does — and does not — show

Independent smoke check

I ran a small independent check using a public Gemma 4 E4B Q4_K_M GGUF on a T4. This was not a reproduction of your Mac, Metal backend, Docker Model Runner, ai/gemma4 artifact, Open WebUI Knowledge implementation, embeddings, retrieval, or citations. It was only a control for the model/native-tool/parser boundary.

The model produced Gemma 4’s native tool syntax with these value types:

• string
• integer
• boolean
• array
• nested object
• null

After converting that native syntax into a normal OpenAI-style tool call, the arguments were a one-layer JSON string that became a dictionary after one parse. For example:

{
  "count": 5,
  "enabled": true,
  "metadata": {
    "priority": 2,
    "source": "colab"
  },
  "optional_ids": null,
  "tags": ["a", "b"],
  "text": "hello"
}

A Knowledge-like call also produced structurally normal arguments:

{
  "count": 1,
  "knowledge_ids": [],
  "query": "maintenance code"
}

I then returned a synthetic tool result:

{
  "results": [
    {
      "content": "The maintenance code is BLUE-7421.",
      "source": "minimal_test.md",
      "score": 1.0
    }
  ]
}

On the next turn, Gemma 4 answered with BLUE-7421 and did not call the tool again.

That supports only a limited conclusion:

Under a simple controlled path, Gemma 4 can emit typed native tool arguments and can consume a tool result in a second turn.

It does not prove that your exact GGUF/template/backend produced the same representation, or that Open WebUI received it unchanged.

Observation	What it supports	What it does not prove
Gemma 4 emitted native tool syntax	Basic tool-call capability exists	Your artifact is identical
Typed values survived normalization	These JSON types can work	DMR normalized them identically
Two-turn result worked	Basic tool-result round trip is possible	Open WebUI reconstructed the transcript correctly
Normal arguments passed an Open WebUI-like mapping step	Expected OpenAI shape can work	Your call had that shape
Artificial double encoding reproduced .items()	One concrete failure mechanism	Your system double-encoded anything

One related observation: some Python llama-cpp-python paths currently return Gemma 4’s native tool block in message.content instead of converting it to message.tool_calls. That is a separate adapter issue, not your exact DMR path, but it illustrates why “the model generated a valid call” and “the application received a valid OpenAI tool call” are different claims. See llama-cpp-python issue #2227.

Where the failure may be happening, and technical notes for the GitHub issue

The relevant data flow

Your Native Knowledge request crosses several boundaries:

Gemma 4 model
→ GGUF artifact and embedded chat template
→ llama.cpp Gemma 4 parser
→ Docker Model Runner OpenAI-compatible response
→ Open WebUI Native middleware
→ Knowledge tool executor
→ retrieval result / error result
→ citation processing
→ tool result added to the conversation
→ Gemma 4 chooses another tool or gives the final answer

Legacy avoids much of this chain. That is why Legacy success is useful evidence, but not a proof that Native should work.

Why the visible .items() error points to a type boundary

In Python, .items() normally expects a dictionary-like mapping.

One Native tool-processing path in Open WebUI v0.10.2 middleware performs the equivalent of:

tool_args = tool_call["function"]["arguments"]
tool_function_params = parse(tool_args)

filtered = {
    key: value
    for key, value in tool_function_params.items()
    if key in allowed_params
}

If parsing returns a dictionary, that is fine.

If parsing succeeds but returns a string, the next .items() can produce exactly:

'str' object has no attribute 'items'

A deliberately double-encoded value is one way to create that situation:

normal:
str → dict

double encoded:
str → str → dict
        ^
        one parse is not enough

That mechanism reproduces the exact exception, but it is not evidence that DMR or Open WebUI actually double-encoded your arguments.

The broader failure family is more useful than one narrow theory:

Failure shape	Example	Likely boundary
Whole arguments value is not an object	One parse still returns a string	serialization/parser/adapter
Outer object is valid, fields have wrong types	"count": "5" or "knowledge_ids": "null"	model output, schema enforcement, coercion
Tool fails first, later processing throws another exception	error object treated as a list of results	error/citation handling
Native model parser changes an array/object into a string	array serialized inside a string	model-specific parser edge case

There are public Open WebUI examples of the second and third shapes:

• In Discussion #20704, query_knowledge_files received "knowledge_ids": "null" and "count": "5", then failed when the string count reached slicing code.
• In Issue #25641, a non-Gemma setup using Llama 3.3 through vLLM passed numeric tool parameters as strings, affecting multiple built-in tools including Knowledge tools.
• In Issue #21070, a tool error result reached citation processing in an unexpected shape, producing a secondary string attribute error.

Those are not necessarily your bug. They show that the visible exception may come from:

1. the original tool arguments,
2. an individual field inside valid arguments, or
3. later processing of an earlier tool failure.

Gemma 4/llama.cpp also has adjacent but non-identical reports:

• array argument serialized as a JSON string
• repetition during a Gemma 4 tool call

Again, these are evidence that this boundary has had edge cases, not proof of the same root cause.

Why 49 calls can happen

Open WebUI’s current CHAT_RESPONSE_MAX_TOOL_CALL_ITERATIONS default is 256. The documentation says this counts sequential tool-calling turns, and that a successful call and a failed call each consume one iteration. It is not simply a “failed retry count.” See Environment Variable Configuration.

So 49 calls can mean either:

Generation/parser loop:
the backend keeps generating the same call before useful execution

Agent recovery loop:
the tool returns an error, the model sees it, changes the query, and tries again

Your UI showing an exception as a tool result makes the second pattern plausible, but the first failed call is needed to tell.

For testing, I would temporarily cap it at a small number such as:

environment:
  CHAT_RESPONSE_MAX_TOOL_CALL_ITERATIONS: "6"

That will not fix the first failure. It only prevents one bad response from consuming a large context and repeatedly calling the same read tool.

Until the loop is understood, I would keep the enabled tools read-only. I would not attach email, shell, file deletion, purchasing, or other side-effect tools to this model/path.

Knowledge attachment is another separate branch

The Knowledge tool does not receive only the text query. Open WebUI must also determine which Knowledge resources are in scope and whether the user can access them.

That creates useful branches:

Tool crashes before retrieval
→ argument/parser/executor path

Tool executes but returns no documents
→ attachment scope, permissions, indexing, or Full Context conflict

Model-attached Knowledge fails but chat-attached Knowledge works
→ custom-model / model-Knowledge injection path

Chat-attached Knowledge fails but model-attached Knowledge works
→ chat resource-scope path

The current Open WebUI Tools documentation also notes that attached tools/resources still depend on user access. Permissions are probably not the first explanation for your exception, because you are seeing a crash rather than a clean empty result, but this matters for later tests and for other readers.

Docker Model Runner’s role

Docker Model Runner provides the OpenAI-compatible endpoint between Open WebUI and llama.cpp. Its current API documentation exposes:

http://localhost:12434/engines/v1/chat/completions

for host-side OpenAI-compatible requests. See the DMR REST API.

The current DMR scheduling/proxy code appears to act mostly as a gateway to the inference backend rather than reconstructing every live Chat Completions response itself. But your installed DMR version, llama.cpp revision, model artifact, and chat template still matter; “OpenAI-compatible” describes the API surface, not a guarantee that every model-specific tool-call representation is preserved perfectly.

The GitHub issue is already good enough that I would not add another speculative comment. A materially useful update would be one of:

• the first raw DMR function.arguments,
• the actual Python type/value immediately before .items(),
• the installed DMR/llama.cpp version,
• a stream:false direct backend response,
• or a model-attached vs chat-attached Knowledge comparison.

The smallest useful checks, as branches rather than a large homework list

Branch 1 — Continue useful testing now

Use four clearly labeled modes:

Mode	Use it for	Do not claim from it
No Knowledge/tools	Base model test	RAG or tools work
Legacy	Temporary classic-RAG control	Native is healthy
Full Context	Short-document direct injection	Retrieval/tool calling works
Native	Full integration test	Base model quality from final answer alone

Open WebUI explicitly says Native Knowledge is not auto-injected, and suggests disabling Native to restore automatic RAG injection or using Full Context to bypass RAG. Full Context is useful for a short reference document, but it is not a successful Native RAG test.

For repeatability, use one tiny Markdown document:

# Test fact

The maintenance code is BLUE-7421.

Prompt:

According to the attached knowledge base, what is the maintenance code?
Do not guess.

Conditions:

new chat
web search OFF
temperature 0 or very low
one document
one Knowledge Base
same model artifact
three runs per mode

Branch 2 — Capture only the first failure

The easy version is to save only:

first tool name
first INPUT
first OUTPUT
first error

Do not collect all 49 calls first. Later calls are recovery behavior layered on top of the original failure.

The precise version is a non-streaming DMR response using the same tool schema:

curl -sS \
  http://localhost:12434/engines/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d @dmr-native-tool-test.json \
  | tee dmr-native-tool-response.json

Then inspect:

choices[0].message.tool_calls[0].function.name
choices[0].message.tool_calls[0].function.arguments
choices[0].finish_reason

Decision tree:

arguments parses once to a dictionary
→ DMR output shape is probably normal
→ inspect Open WebUI parsing / Knowledge injection

arguments parses once to another string
→ whole-value double encoding
→ directly matches one `.items()` mechanism

outer dictionary is normal, but count/IDs are strings
→ field-level type drift or missing coercion

tool_calls is absent and native Gemma tokens appear in content
→ model-specific parser did not classify the call

first turn is normal, second turn breaks
→ tool-result transcript / multi-turn template path

Branch 3 — Separate a generic tool from Knowledge

A Knowledge call is a relatively complicated test. A typed read-only echo tool is a cleaner protocol check.

Example expected arguments:

{
  "text": "hello",
  "count": 3,
  "enabled": true,
  "tags": ["a", "b"],
  "metadata": {
    "source": "test"
  },
  "optional_ids": null
}

Interpretation:

typed toy tool also fails
→ Native tool path generally

typed toy tool works, Knowledge fails
→ Knowledge-specific scope/executor/result path

DMR direct works, Open WebUI fails
→ Open WebUI boundary

both work once, but multi-turn fails
→ transcript/template/loop boundary

This is why 7 + 3 is not enough: it tests ordinary text generation unless a calculator tool actually exists and is called.

Branch 4 — Score each stage, not only the final answer

Stage	Record
Tool was available	Yes/No
Correct tool selected	Yes/No
Arguments were parseable	Yes/No
Field types were correct	Yes/No
Tool executed	Yes/No
Relevant chunks returned	Yes/No
Model used the result	Yes/No
Citation produced	Yes/No
Duplicate calls	Count
Final answer correct	Yes/No

This prevents an integration bug from becoming a false “bad model” score.

AnythingLLM, reproducibility, and the longer-term platform decision

AnythingLLM

AnythingLLM is a reasonable comparison, especially if you need a supported Native path now. I would still run it beside Open WebUI first rather than migrating the whole setup.

Keep constant:

same DMR endpoint
same Gemma artifact
same tiny document content
same expected answer
same temperature
web search OFF

Allow to differ:

frontend
ingestion/chunking
embedding/vector store
retrieval prompt
citation system
agent/tool loop

Interpretation:

AnythingLLM works, Open WebUI Native fails
→ you have a practical alternative path
→ this does not identify Open WebUI as the root cause

both Native paths fail
→ shared model/artifact/backend compatibility becomes more suspicious
→ still not proof of the same failure

both work
→ choose by workflow, observability, and maintenance preference

AnythingLLM’s current documentation says Native tool calling is enabled by default for supported local providers and can be disabled per provider if it misbehaves. It also defaults SDK retries to 0 to avoid duplicate requests to local models. See AnythingLLM Configuration.

Its current Native tool-calling changelog also describes a maximum of 10 tool calls per response as a runaway-loop safeguard. See AnythingLLM v1.11.1.

The caution is that connecting DMR through Generic OpenAI is deliberately described as a developer-focused configuration. “The endpoint accepted a chat request” does not by itself confirm correct tools, streaming, transcripts, or RAG behavior.

Make the test bench versioned

The stack is now large enough that this is effectively small personal LLMOps:

model
artifact / quantization
chat template
llama.cpp backend
Docker Model Runner
Open WebUI
embedding model
Knowledge configuration
web search
tools

Record at least:

date
Open WebUI version or image digest
Docker Desktop version
DMR version
llama.cpp version
model name, tag, and digest
quantization
embedding model
Native/Legacy/Full Context
Knowledge attachment method
web search on/off
stream true/false
temperature

docker model status can show the running inference engine and llama.cpp version in current Docker documentation.

Do not compare model scores across two runs as if they were the same experiment when the integration stack changed. Mark that as a new environment generation.

A compact record could look like:

run:
  date: 2026-07-09
  open_webui: v0.10.2
  dmr: <version>
  llama_cpp: <version>
  model: ai/gemma4
  model_digest: <digest>
  quant: Q4_K_M
  embedding: all-MiniLM-L6-v2
  mode: Legacy
  knowledge_attachment: model
  web_search: false
  stream: true
  temperature: 0

Bottom line

I would not switch away from Open WebUI just because this Native path is currently failing.

I would use this default route:

continue the test bench
→ base-model tests without tools/Knowledge
→ Legacy as the temporary RAG control
→ Native as a separate integration test
→ AnythingLLM as a parallel comparison only if needed

The most likely general category is a protocol/type boundary inside the Native tool path, possibly combined with a retry loop. The current evidence does not justify naming a single guilty component.

The one missing artifact that would change the diagnosis most is:

the first failed, non-streaming tool call — especially the raw function.arguments value before Open WebUI’s repeated agent loop.

Until that appears, I would avoid rebuilding the Knowledge Base, replacing the embedding model, changing several components at once, or assuming that a frontend migration is guaranteed to fix it.

[alaskaav8r]

thank you @John6666 I began testing yesterday and will report results once i have a good handle on this.

Tim

[alaskaav8r]

ok testing so far,

normal chat = pass

summarization: I had all tools and knowledge bases OFF, I am on legacy mode (already a github bug report #26880). Attaching a document via the chat window and asking to summarize the document = pass, then open new chat window and upload same document and ask to extract only warnings and safety notes in the document i just uploaded = Fail. Even with strong sys pr for the model it still failed this test and would pull information from other docs in the knowledge base, specifically my nikon manual. The only way to stop this was to specifically state the name of the document in the prompt, then it worked correctly. This also was tested in Native, Default and Legacy modes for function calling. Submitted a new bug report for this issue github #27073.

Moving on to rag test but not sure if it wouldn’t be wasting my time doing that since i already have the github bug #26880

[John6666]

I think your test results may have uncovered another important boundary:

---

I would keep going with the RAG tests. This was not wasted work.

The new result appears to expose a different boundary from the one in Open WebUI issue #26880:

Issue	Main boundary being tested
#26880	Native tool calls, argument parsing, tool execution, and retry loops
#27073	Which document is retrieved and used inside a bound multi-file Knowledge Base

The important clue is this:

“Use only the document I just attached”
→ content from the Nikon manual still appeared

“Use only the document named Turning Point Test”
→ the answer became correctly scoped

That suggests the system can use the intended document when it receives a strong filename anchor, but the phrase “the document I just attached” may not be turning into a strict file-level retrieval filter.

In other words:

A document being available to the model is not the same as retrieval being restricted to that document.

This looks less like a simple “the model ignored the instruction” failure and more like a retrieval-scope or source-selection boundary.

The practical route I would use

I would now split the Knowledge tests into four separate lanes:

Test lane	Setup	What it measures
Exact document reading	One file in Full Context	Can the model understand that specific document?
Strict RAG control	One file in one Knowledge Base	Can indexed retrieval find and use the correct content?
Collection RAG	Multiple files in one Knowledge Base	Can retrieval select the correct source among distractors?
Native Knowledge	Native function calling and Knowledge tools	Can the full tool-selection, file-discovery, retrieval, and multi-turn path work?

For now, my default route would be:

1. Keep the exact filename in the prompt as a practical workaround and test condition.
2. Keep a one-file-per-Knowledge-Base control for strict source-isolation tests.
3. Keep the mixed Knowledge Base as a separate collection-retrieval stress test.
4. Do not change the embedding model, chunk size, reranker, model artifact, and prompt all at once.
5. Score the source, not only whether the final answer sounds correct.

A useful pass condition is:

correct answer
AND correct source
AND excluded sources absent

A plausible answer taken from the wrong manual should still count as a failure.

Why this looks like a retrieval-scope boundary

The retrieval process has several separate stages

A simplified RAG path looks like this:

document is available
→ retrieval scope is selected
→ candidate chunks are searched
→ candidates are ranked
→ selected context is added to the prompt
→ the model uses or ignores that context
→ answer and citations are generated

A document can be successfully uploaded and attached while a later stage still searches a broader collection than the user intended.

Open WebUI’s Knowledge documentation distinguishes two relevant modes:

• Focused Retrieval searches for chunks that are semantically relevant to the user’s query.
• Full Context inserts the entire document directly, without chunking or semantic search.

For an attached Knowledge Base, the supported scope is the attached collection. That does not necessarily mean that natural-language phrases such as:

the file I just attached
this document
the document above

become a hard filter equivalent to:

file_id = <specific file>
source = <specific filename>
path = <specific path>

So this instruction:

Use only the document I just attached.

may be applied mainly during answer generation, after retrieval has already searched the bound collection.

That creates a possible flow like this:

User-intended scope:
Turning Point Test only

Actual retrieval scope:
the whole bound Knowledge Base

Query concepts:
warnings / safety / danger / caution

Strong semantic match:
warning-heavy chunks in the Nikon manual

Context given to the model:
chunks from the intended file plus Nikon chunks

Final answer:
plausible safety information from the wrong source

We do not yet have the raw retrieval candidate list, so this is a working explanation rather than a confirmed internal trace. But it fits the observations reasonably well.

Why the filename result matters

Explicitly naming the file may be helping at one or more stages:

1. The filename may become a strong term in the retrieval query.
2. It may help the model associate retrieved chunks with the intended source.
3. Retrieval may remain broad, but the model may filter the context more accurately during answer generation.
4. Citation selection may become better aligned with the requested source.

The current result does not tell us which one happened.

So I would phrase the finding as:

Naming the file is a useful workaround and a strong diagnostic clue, but it does not yet prove that retrieval itself was restricted to that file.

Why “warnings and safety notes” was a good test

That wording creates a strong semantic collision.

A product manual containing many sections labeled:

WARNING
CAUTION
DANGER
SAFETY

is likely to be highly relevant to a semantic query about warnings and safety notes, even when it is not the document the user intended.

This makes your prompt a useful adversarial RAG test:

weak scope reference:
“the document I just attached”

strong distractor concepts:
“warnings and safety notes”

distractor document:
manual with many warning sections

It is exactly the kind of test that can reveal whether the retrieval system follows a source boundary or simply chooses the most semantically similar chunks in the collection.

A small test matrix that keeps the results interpretable

Use tiny documents when testing the boundary

The real documents are useful, but a tiny synthetic pair can make the result easier to interpret.

Target document

# Turning Point Test

The maintenance code is BLUE-7421.

This document contains no warnings, cautions, danger notices, or safety instructions.

Distractor document

# Nikon Safety Manual

WARNING: Example warning text.

CAUTION: Example caution text.

DANGER: Example danger text.

Then use a small matrix:

Case	Configuration	Purpose
A	Target file only, Full Context	Exact reading control
B	Target file only, one-file KB	Indexed single-source RAG control
C	Target + distractor in one KB, no filename	Collection-scope collision test
D	Same mixed KB, exact target filename included	Filename-anchor test
E	Same mixed KB, deliberately wrong filename	Source-attribution sensitivity
F	Native file discovery and single-file reading	Exact-file agentic test after #26880

Keep these constant:

same model and quantization
same embedding model
same retrieval configuration
same attachment method
web search OFF
low or zero temperature
fresh chat
same question

A few fresh-chat repetitions are useful because a single pass may be accidental.

Suggested interpretation

Full Context fails
→ model, extraction, context, or instruction-following problem

Full Context passes, one-file KB fails
→ retrieval/indexing/model-uptake problem

one-file KB passes, mixed KB fails
→ collection source-selection problem

filename fixes the mixed KB
→ filename/source anchoring is important

wrong filename changes the source or answer
→ retrieval-query or attribution sensitivity

both files appear in citations
→ distractor context entered or survived into answer generation

answer is correct but citation is wrong
→ attribution failure rather than answer-generation failure

Score each stage separately

A useful result sheet could contain:

Stage	Record
Intended document available	Yes/No
Intended document retrieved/cited	Yes/No
Distractor document retrieved/cited	Yes/No
Answer content correct	Yes/No
Answer source correct	Yes/No
Filename was required	Yes/No
Reproducible in fresh chats	Yes/No

This avoids turning several different problems into one general “RAG failed” result.

It also prevents a particularly dangerous false pass:

The answer sounds correct,
and the citation is real,
but both belong to the wrong product or document.

That failure can be harder to notice than an obvious hallucination.

Practical controls now, and the stronger Native route later

What can be used now

1. Explicit filename

For ordinary use, explicitly naming the intended file is a reasonable workaround:

Using only the document named “Turning Point Test”,
extract its warnings and safety notes.
Do not use any other document.

I would keep this as a controlled test condition, not assume it creates a guaranteed hard retrieval filter.

2. One file per Knowledge Base

For a strict indexed-source control:

KB: Turning Point Test
└── Turning Point Test.pdf

This is less convenient than a large mixed collection, but it gives a much cleaner boundary.

If the one-file KB works and the mixed KB fails, that is useful evidence that the problem is source selection within the collection rather than document extraction or basic retrieval.

3. Full Context for short documents

For a short document where exact reading matters more than retrieval efficiency, Full Context is a useful control.

According to the Open WebUI Knowledge documentation, Full Context:

• injects the complete file,
• performs no chunking,
• performs no semantic search,
• and does not require the model to call a Knowledge tool.

That makes it useful for:

short procedures
small policy documents
style guides
single-document extraction tests

But it should not be counted as a successful RAG or Native Knowledge test. It bypasses those mechanisms.

4. Keep the mixed KB as a stress test

A mixed KB is still valuable. It measures a harder and more realistic task:

Can the retriever find the intended source when several plausible documents are available?

That is a different capability from reading one already-selected document.

The stronger Native route later

Open WebUI’s current Knowledge tools support more explicit file-navigation steps, including:

• search_knowledge_files to search by filename,
• query_knowledge_files for semantic retrieval,
• grep_knowledge_files with a file_id single-file mode,
• view_file to read a particular file or line range.

The documentation describes a typical agentic sequence like:

query_knowledge_files
→ locate the relevant document

grep_knowledge_files(file_id=...)
→ search within that specific file

view_file
→ read the surrounding lines

There is also the experimental kb_exec path, which can address documents by path, filename, or file ID and use operations such as find, grep, cat, and sed.

That is conceptually stronger for an exact-file request because the model can first identify a file and then use a mechanical file boundary.

However, in your environment this is not the immediate route while #26880 is still blocking the Native tool path.

So I would currently treat the modes as:

Legacy / Default
→ practical control and workaround

Full Context
→ exact short-document reading control

Native
→ intended agentic file-navigation path,
   to revisit after the tool-call issue is resolved

A useful rule for organizing Knowledge Bases

Group documents by safe retrieval scope

A useful Knowledge Base boundary is not merely:

same topic
same folder
same user
same project

A stronger question is:

Would it be safe and correct for these documents to be retrieved together for the same query?

Documents that may need separate Knowledge Bases include:

• manuals for different products or machines,
• current and obsolete procedures,
• draft and approved instructions,
• contracts for different customers,
• regulations for different regions,
• production and test environments,
• aircraft or equipment variants,
• old and current software-version documentation.

For example:

Nikon manual
Turning Point installation document
unrelated maintenance manual

may all be legitimate documents for the same user, but they are not necessarily safe retrieval peers for every question.

A practical rule is:

Documents that may be searched together safely
→ same Knowledge Base

Documents that must not contaminate each other
→ separate Knowledge Bases or an explicit file-level filter

Three scopes to keep separate

Access scope

Which Knowledge Bases or files is the user allowed to access?

Retrieval scope

Which accessible documents are searched for this request?

Answer scope

Which retrieved documents does the model actually use in the answer?

A system can enforce access scope correctly while still retrieving the wrong accessible document.

Your testing already found an encouraging boundary: unrelated Knowledge Bases that were not bound to the model did not appear. The apparent contamination remained inside the bound collection. That makes this look more like retrieval/source selection than unrestricted cross-KB access.

A multi-file Knowledge Base should therefore be treated as a collection-level retrieval boundary, not automatically as a hard per-file isolation boundary.

Bottom line

I would continue the RAG tests.

The new result is useful because it reveals a failure that final-answer scoring alone could easily miss:

The system found plausible information from a real source, but it was the wrong source for the user’s intended scope.

For the next phase, I would use this default structure:

Exact reading of one short document
→ Full Context

Strict indexed single-source control
→ one file per Knowledge Base

Collection retrieval
→ mixed Knowledge Base, with source correctness scored separately

Exact file navigation
→ Native filename/file_id tools after #26880 is resolved

And I would keep the filename-explicit prompt as both:

a practical workaround
and
a diagnostic condition

The most important lesson from this test is:

“Use only this document” is not necessarily equivalent to a retrieval-time file filter.

That is exactly the kind of boundary a useful local-AI test bench should detect.

[alaskaav8r]

John i haven’t had time to go through your response yet but i finished the test as best I could. Few quick questions, when testing “safety” do I need to be in normal everyday settings, or in the settings for test mode. Also for the stability test, do i have to wait and test other items or can I just do it as a repeat immediately following the appropriate section so i’m not chasing down settings?

I am attaching a .pdf that outlines a testing path that I used based on your input. Much of it I didn’t understand at first but learning fast.

My frustration is testing the model in this broken condition, meaning that open webUI is one of my major problems.

testing document

[John6666]

Hi. For now, I think this is quite good, but after looking into it, I found a few things worth correcting:

---

For your two questions first:

• Safety: I would test both the normal-use configuration and the isolated diagnostic configuration, but record them as two separate tests. They answer different questions.
• Stability: You do not need to wait. Repeating a test immediately, before changing any settings, is the cleanest first stability check.

I would also keep the results you already recorded. Most of the observations are useful. The main corrections are about which boundary a test actually reached, and therefore how some Pass/Fail results should be labeled.

I marked the changes directly in a revised copy here:

Revised Gemma 4 test sheet with marked corrections

The red REVISED notes mainly correct test boundaries, labels, or result interpretation. They are not intended to discard the observations you recorded.

The most important corrections

Original interpretation	More precise interpretation
localhost:3000 is a backend-direct test	It is an Open WebUI API test without the browser UI
A returned tool_calls field must also execute the tool	Call generation, parsing, execution, result return, and final answer are separate stages
Native ON + Full Context is a pure Full Context test	It can be a mixed condition if Knowledge tools are still available
The answer to “What can you access?” proves actual permissions	It tests self-description; real retrieval/tool behavior is stronger evidence
One repeated run is a general Stability result	Same-chat, fresh-chat, and post-restart stability test different things

The overall result still looks important:

Legacy / automatic retrieval
→ document questions can work

Full Context / direct document reading
→ document questions can work

Native Knowledge tools
→ repeatedly fail on the current integration path

Generic Native tools
→ different tools appear to reach different stages

That narrows the problem more usefully than simply concluding that Gemma 4 is bad at RAG or tools.

My recommended default path

I would proceed like this:

1. Keep all existing observations.
   Why: the outputs and error behavior are useful evidence even where the original Pass/Fail label needs refinement.

2. Do not change the model, embedding model, runtime, prompt, and tool settings together.
   Why: if several variables change, an improvement or regression cannot be attributed to one layer.

3. Repeat the ambiguous test immediately with the same settings.
   Why: this checks whether the result is stable before conversation history, restarts, or configuration changes introduce another variable.

4. Repeat the same cell once in a fresh chat.
   Why: this separates same-conversation state from independent reproducibility.

5. Retest only the ambiguous cells rather than rebuilding the entire sheet.
   Why: the highest-value remaining questions are concentrated around the API boundary, Native + Full Context, tool execution, and Safety.

6. Keep :3000 and :12434 as separate test lanes.
   Why: they pass through different components and therefore help locate the failing boundary.

7. Keep Legacy as a working control, not as proof that Native is fixed.
   Why: it shows that the documents, indexing, retrieval, and model can work through at least one path, while Native issue #26880 remains a separate integration failure.

What the current results already establish

Legacy and basic document use are not completely broken

Your results show that, with Native disabled, the system can:

• retrieve the water-pump fuse location and rating;
• correctly say that a replacement part number is absent;
• answer questions from the beginning, middle, and end of the longer Z8 manual;
• use both Focused Retrieval and direct Full Context successfully in at least some controlled cases.

That means the following are not all universally broken:

document extraction
index creation
embedding availability
basic retrieval
context injection
Gemma’s ability to answer from supplied text

It does not prove that every document and retrieval configuration is correct, but it gives you a useful working control.

Native Knowledge remains the strongest repeated failure

The Native path fails across:

• a short test document;
• a long manual;
• questions from different document positions;
• information that exists in the document;
• information intentionally absent from the document;
• query_knowledge_files;
• and previously kb_exec.

That is consistent with the boundary already documented in Open WebUI issue #26880, which is still open at the time of writing.

The useful conclusion is not necessarily:

Gemma cannot retrieve knowledge.

A more precise conclusion is:

The current Gemma 4 + Docker Model Runner + Open WebUI Native Knowledge integration does not complete the retrieval/tool loop reliably, while the Legacy retrieval path can use the same documents successfully.

The generic tools produced more than one failure pattern

Your generic-tool results are also useful because not every tool behaved identically:

add_two_numbers
→ attribute error

lookup_fake_order_status
→ attribute error

get_current_time
→ returned a time, but not the desired time zone

That suggests the broad Native path is not simply “nothing works.”

Different tools may differ in:

• argument schema;
• argument types;
• whether parameters are required;
• tool implementation;
• result shape;
• how the call is parsed;
• or how the result is returned to the model.

So I would avoid the broader conclusion:

Gemma 4 cannot call tools.

The narrower observation is stronger:

Some tool calls appear to be generated, but different calls fail or complete at different integration stages.

The API boundary correction

localhost:3000 is still inside Open WebUI

The test sheet currently uses:

http://localhost:3000/api/chat/completions

That is Open WebUI’s chat-completions API. It bypasses the browser interface, but it does not bypass Open WebUI itself.

A more accurate label is:

Open WebUI API test without the browser UI

Docker Model Runner’s OpenAI-compatible host endpoint is normally:

http://localhost:12434/engines/v1/chat/completions

Docker documents the host base URL and endpoint in its Model Runner REST API reference.

So the useful boundary comparison is:

Test	Endpoint	Main layers still involved
Open WebUI browser chat	Open WebUI UI	UI, Open WebUI middleware, tools, agent loop, DMR
Open WebUI API	localhost:3000/api/chat/completions	Open WebUI middleware, model/tool configuration, DMR
DMR direct API	localhost:12434/engines/v1/chat/completions	DMR, llama.cpp runtime, model/template/parser

This does not make the :3000 result unimportant.

Seeing Gemma-native tool syntax appear as ordinary content through the Open WebUI API is a valuable observation. It tells us that the structured call was not completed correctly by the time the response reached that API boundary.

But it does not yet show whether the raw syntax originated because:

• Docker Model Runner returned plain content;
• the llama.cpp parser did not convert it;
• Open WebUI received a structured call but transformed it incorrectly;
• the model preset did not include the expected tool contract;
• or a streaming/non-streaming path behaved differently.

Comparing the raw :12434 response with the raw :3000 response using the same model, prompt, tool schema, and streaming mode would narrow that boundary considerably.

A tool call is several tests, not one test

A complete tool round-trip has multiple stages

A real tool test is better scored as a sequence:

1. A tool definition or server-side tool ID is available
2. The model requests the intended tool
3. The response parser creates a structured tool call
4. The application recognizes the call
5. The tool executes
6. The result is returned to the model
7. The model uses the result in its final answer
8. A later tool call can still work in the same conversation

A result can pass one stage and fail the next.

For example:

Gemma emits the correct function and arguments
→ model call-generation pass

A real tool_calls field appears
→ parser/serialization pass

No tool executes
→ execution or orchestration not yet demonstrated

The tool executes but the model ignores its result
→ result-passback or model-uptake failure

tool_calls does not always mean automatic execution

In an OpenAI-compatible protocol, the normal client-owned flow is:

model returns tool_calls
→ client executes the requested function
→ client sends a tool-result message
→ model produces the final answer

Therefore, a direct API response containing a valid tool_calls array may be a successful first turn rather than a failed complete loop.

Whether Open WebUI executes the call server-side depends on how the tool was supplied.

Open WebUI’s API endpoint documentation distinguishes:

• Open WebUI-managed tools selected using tool_ids;
• and caller-provided OpenAI-style definitions supplied in a tools array.

If the curl request contains only:

{
  "model": "gemma-4-clone",
  "messages": [
    {
      "role": "user",
      "content": "Use the add_two_numbers tool to add 17 and 25"
    }
  ],
  "stream": false
}

then the request itself does not show which tool definition was supplied.

The custom model may add tools internally, but for a reproducible API test it is better to record the actual contract explicitly:

server-side Open WebUI tool
→ include and record tool_ids

client-owned tool
→ include and record the complete tools schema
→ execute returned call in the client
→ send the result back in the next request

Suggested result columns

Stage	Result
Correct tool selected	Pass/Fail
Correct argument values	Pass/Fail
Correct argument types	Pass/Fail/Not observable
Structured tool_calls produced	Pass/Fail
Tool actually executed	Pass/Fail/Not tested
Result returned to model	Pass/Fail/Not tested
Final answer used result	Pass/Fail/Not tested
Second call succeeded	Pass/Fail/Not tested

This avoids converting “not tested” into “failed.”

Streaming and non-streaming need different handling

stream: false

This normally returns one JSON response, so tools can be inspected directly in:

choices[0].message.tool_calls

This is the easier diagnostic starting point.

stream: true

This returns an event stream rather than one ordinary JSON document. Tool-call data may be split across several deltas and must be accumulated before comparison.

Therefore, piping the entire streaming response directly into:

python3 -m json.tool

does not reliably test whether a streamed tool call was valid.

A better comparison is:

same endpoint
same model
same prompt
same tools/tool_ids
same sampling settings

only difference:
stream false vs stream true

Then record separately:

Check	Non-stream	Stream
Tool name reconstructed
Arguments reconstructed
Types preserved
Tool executed
Result returned
Final answer completed

Interpretation:

non-stream fails and stream fails
→ shared model/parser/tool-contract path remains suspect

non-stream passes and stream fails
→ streaming aggregation/parser path becomes more likely

both return structured calls but neither executes
→ execution may not have been part of the direct API test

browser chat executes but direct API only returns tool_calls
→ the browser/Open WebUI agent may be performing the orchestration

How to make Full Context a clean control

Full Context and Native Knowledge are different paths

Open WebUI’s Knowledge documentation describes Full Context as:

• complete document injection;
• no chunking;
• no semantic search;
• available regardless of the Native Function Calling setting;
• no Knowledge tool call required.

Its Tools documentation also warns against using Full Context with Knowledge tools, because Full Context bypasses the vector-store retrieval path.

That means this configuration can be ambiguous:

Native Function Calling = ON
Full Context = ON
Knowledge tools still available

If the model unnecessarily calls query_knowledge_files and that call fails, the result does not by itself prove that the direct Full Context document injection failed.

It may instead mean:

the document was already in context
but
the model selected an unnecessary broken tool path

Split it into two cleaner tests

Exact Full Context reading

Full Context = ON
Knowledge retrieval tools unavailable
Web search = OFF
Custom tools = OFF

Question being tested:

Can the model answer correctly when the complete document is already supplied directly?

Native Knowledge retrieval

Full Context = OFF
Native Function Calling = ON
Knowledge tools = ON
Web search = OFF
Unrelated custom tools = OFF

Question being tested:

Can the model select a Knowledge tool, retrieve the correct passage, receive the result, and answer?

The mixed configuration can still be tested as a daily-use compatibility case, but it should not replace either clean diagnostic control.

Safety: test normal use and isolated use separately

Yes, Safety belongs in both configurations

Normal-use Safety

Use the settings you actually expect to run daily:

normal Knowledge Base
normal tools
normal web-search setting
normal model preset
normal user permissions

This measures deployment behavior:

• Does unrelated retrieved material influence the answer?
• Does the system describe its capabilities misleadingly?
• Does it cite sources that do not answer the question?
• Does it confuse product documentation with a statement about its own access?

Your Legacy result—where the question about accessible files triggered retrieval of Open WebUI product documentation—is useful evidence of exactly this kind of source-selection problem.

Isolated diagnostic Safety

Use a minimal configuration:

fresh chat
web search OFF
one known Knowledge Base
known tool list
known permissions
no unrelated sources

This measures whether the model and minimal integration can stay within a clearly described boundary.

These should be recorded separately:

Test	Purpose
Safety — normal use	Is the intended deployment safe and understandable?
Safety — isolated diagnostic	Does the minimal system invent capabilities or sources?

A self-report is not a permission audit

The prompt:

What files can you access?

is useful, but the answer is still a model-generated self-description.

The model may not have direct introspection into every effective setting, permission, model attachment, or tool authorization.

A stronger test uses canaries:

Accessible attached document:
ACCESS-CANARY-BLUE-7421

Unattached or inaccessible Knowledge Base:
DENIED-CANARY-RED-9184

Then test whether it can actually retrieve each value.

A stronger Pass condition is:

accessible canary retrieved correctly
AND
inaccessible canary not retrieved
AND
tool/citation matches the actual source

This distinguishes:

“I believe I can access X”

from:

“I actually retrieved X through the permitted path.”

For the current minimal Safety result, I would use a label such as:

Better scoped self-description: Pass
Actual permission boundary: Not yet verified

Stability: repeat now, but distinguish three meanings

1. Same-chat immediate repeat

Repeat the same request immediately in the same conversation.

This tests:

• accumulated conversation state;
• whether a second tool call works;
• whether prior tool output corrupts the transcript;
• whether retry behavior changes after the first failure.

2. Fresh-chat immediate repeat

Open a new chat and repeat the identical test without changing settings.

This tests:

• independent reproducibility;
• whether the result depends on previous messages;
• whether a clean transcript changes the outcome.

3. Post-restart or post-update repeat

Repeat later after restarting or changing a version.

This tests:

• operational stability;
• persistence;
• version regressions;
• runtime initialization differences.

You do not need to wait before performing the first two.

A low-effort pattern would be:

run test once
→ repeat once in the same chat
→ repeat once in a fresh chat
→ record all three before changing settings

Then preserve one small subset for later post-restart regression testing.

Environment fields that would make future comparisons easier

The environment page is already much more useful than an informal description.

I would add a few fields at the individual-run level, because settings change between sections:

run ID and timestamp
same chat or fresh chat
Open WebUI version/digest
Docker Desktop version
Docker Model Runner version
actual llama.cpp engine revision
model tag and digest
quantization
embedding model
Native or Legacy
Full Context ON/OFF
stream true/false
Knowledge Base name
attachment method
enabled built-in tools
enabled custom tools
tool ID or complete tool schema
temperature/sampling settings

Docker documents:

docker model status
docker model status --json

for inspecting the running Model Runner status; see the docker model status documentation.

The useful principle is:

Record the actual runtime revision before updating it.

That gives an issue maintainer or future tester a reproducible version boundary without requiring you to update everything immediately.

Claims I would keep provisional

I would avoid these broad conclusions for now:

The backend-direct test failed.
Open WebUI’s parser is definitely the root cause.
Full Context itself failed.
The Safety permission test passed.
Gemma 4 cannot call tools.
A tool_calls response should automatically execute the tool.

More defensible wording would be:

The Open WebUI API response exposed Gemma-native tool syntax as content.

The result is consistent with a parser, adapter, tool-contract, or orchestration boundary, but a DMR-direct comparison is still needed to locate the exact stage.

The mixed Native + Full Context test failed after a Knowledge tool call; it does not yet prove that direct Full Context injection failed.

The isolated Safety answer was better scoped, but actual access should be verified through behavior.

Some tools reached different stages, so schema and execution-path differences may matter.

A structured tool_calls response demonstrates call serialization, not necessarily execution or a complete round-trip.

Bottom line

I would not discard or restart the test sheet.

The observations already give a useful picture:

Legacy retrieval
→ working control

Full Context document reading
→ working control when isolated

Native Knowledge
→ repeated blocking failure

Generic Native tools
→ different calls reach different stages

Multi-file source scoping
→ separate open issue #27073

The best next step is not a complete rerun. It is to:

correct the labels
keep the original observations
repeat the ambiguous cells
separate :3000 from :12434
separate structured call generation from execution
separate Full Context from Knowledge-tool retrieval
run Safety in both normal and isolated configurations
run Stability immediately in same-chat and fresh-chat forms

So the strongest current conclusion remains:

The documents and the model can work through Legacy retrieval and direct document context, while the Native Knowledge/tool integration remains the main blocking path.

That is a much narrower—and much more actionable—result than “Gemma 4 failed.”

[alaskaav8r]

@John6666

• Safety: I would test both the normal-use configuration and the isolated diagnostic configuration, but record them as two separate tests. They answer different questions.

okay first question: Safety: I would test both the normal-use configuration and the isolated diagnostic configuration, but record them as two separate tests. They answer different questions. As far as testing this in normal config that is easy, You say they answer different question, when in normal config, what is the question that it is answering?

Now in diagnostic config, there are so many different variations, web search On or OFF, knowledge bases ON or OFF, full context ON or OFF, Native Function calling On or OFF, Built in tools ON or OFF, Custom Tools ON or OFF, and finally specifically what question does this configuration answer in regards to safety?

[alaskaav8r]

@John6666

• Stability: You do not need to wait. Repeating a test immediately, before changing any settings, is the cleanest first stability check.

• Stability: You do not need to wait. Repeating a test immediately, before changing any settings, is the cleanest first stability check.

Now for stability you mentioned: “repeat one RAG question after changing nothing”. Since my RAG test is a 4 way matrix of settings, should i only repeat the RAG question in one of those steps and if so which one, or should i repeat the RAG question in each of the 4 steps (configurations)?

[alaskaav8r]

In response to recording parameters, you mention “knowledge tools exposed”, with Native mode active claude shows i have the following tools available to be used:

1. list_knowledge / list_knowledge_bases — discover what KBs are attached or available
2. query_knowledge_files — semantic/conceptual search within attached files (the main one used for most questions)
3. query_knowledge_bases / search_knowledge_bases — search across KBs by name/description when none are explicitly attached to the model
4. view_file / view_knowledge_file — read a specific file’s full content directly, rather than searching fragments
5. grep_knowledge_files — exact string/identifier search within files
6. kb_exec — an experimental filesystem-style interface (ls, tree, grep, cat) over your knowledge, enabled via ENABLE_KB_EXEC=True, only active in Native mode

How do i record that parameter, am I looking to see which specific tool it called?

[alaskaav8r]

In recording Custom Tool IDs / Schemas:

I’m assuming I will need to run a python script to get this information, or is this the tool name and code i find in the open webUI Tools section i.e.

add_two_numbers

import os
import requests
from datetime import datetime
from pydantic import BaseModel, Field

class Tools:
def init(self):
pass

def add_two_numbers(self, a: int, b: int) -> int:
    """
    Add two numbers together.
    :param a: The first number.
    :param b: The second number.
    :return: The sum of a and b.
    """
    return a + b

[John6666]

Well. Hmm… fair point. It looks like there were still quite a few things I should have clarified:

---

The four questions in posts #8–#11 point to the same underlying problem in my earlier wording:

I did not separate what was configured, what was expected to be available, and what actually happened during the run clearly enough.

I would not rebuild the test sheet from scratch. I would keep the results and add two small sections:

A. Pre-run configuration
   What was attached, enabled, selected, and expected to be available?

B. Per-run execution trace
   What did the model actually call, what executed, what returned,
   and what did the final answer use?

Once those are separate, most of the ambiguity around Safety, Stability, Knowledge tools, and Custom Tool schemas becomes much easier to handle.

The marked revision is still here:

Revised Gemma 4 test sheet with marked corrections

Direct answers to #8–#11

Post	Direct answer
#8 — Safety configuration	Do not create one universal “diagnostic configuration.” Keep one normal-use Safety test, then use a few small diagnostic lanes, each designed to test one specific safety boundary.
#9 — Stability repetition	Repeat the full four-way matrix when validating the matrix itself. For routine regression testing afterward, keep a smaller set of representative sentinel tests.
#10 — Knowledge tools exposed	Record KB binding, active document/retrieval mode, expected tool availability, and actual tool use separately. A KB being attached is not the same as a tool being injected or called.
#11 — Custom Tool IDs and schemas	You do not need a Python extraction script for the basic record. Start with the Open WebUI Tool ID/name, function name, signature, type hints, and expected result. Capture the exact generated JSON schema only when debugging schema or argument-type problems.

The common rule is:

Configured, attached, available to the model, called, executed, returned, and used are different states.

Recommended default path

I would make the smallest possible change to the current test sheet:

1. Keep all existing observations.
   Why: the responses, citations, errors, and tool names are still useful evidence. Most corrections concern the label or test boundary, not the observation itself.

2. Add one Pre-run Configuration table per test section.
   Why: otherwise it is difficult to distinguish “the model ignored an available tool” from “the tool was never available in that run.”

3. Add one Per-run Execution Trace table.
   Why: a tool definition can be available without being called, and a call can be generated without being parsed, executed, returned, or used.

4. Do not combine every Safety toggle into one giant matrix.
   Why: a large matrix may show that something failed, but not which safety property was being tested.

5. Repeat the complete RAG matrix once, then keep a smaller regression set.
   Why: validating the original matrix and performing lightweight future stability checks are different jobs.

6. Treat the model’s own tool/access description as a clue, not the configuration record.
   Why: the model may not be able to inspect every effective Open WebUI setting, user permission, attachment state, or actual request payload.

7. Use the exact UI label visible in Open WebUI v0.10.2 when recording settings.
   Why: the current Open WebUI documentation may describe controls or names added after the version being tested.

A compact status vocabulary would also help:

Pass
Fail
Blocked
Not tested
Not observable
Mixed / ambiguous condition

For example:

Native integration failed before the Safety behavior could be observed
→ Blocked

No raw request payload was captured
→ Tool injection: Not observable

The model produced a tool call but no client executed it
→ Call generation: Pass
→ Execution: Not tested

Post #8 — What should the Safety tests actually measure?

Normal-use Safety and diagnostic Safety answer different questions

Normal-use Safety

Use the configuration you actually expect to run day to day.

That may include:

your normal Function Calling mode
your normal Knowledge Base attachments
your normal File Context / retrieval mode
your normal web-search setting
your normal tools
your normal permissions
your normal model preset and system prompt

The question is:

In the real deployment configuration, does the whole system remain within the expected source, access, capability, and action boundaries?

This tests the deployed stack, not only Gemma:

Gemma
+ Docker Model Runner
+ Open WebUI
+ attached Knowledge
+ retrieval
+ web search
+ tools
+ permissions
+ citations

Useful checks include:

• Does it answer from the intended source?
• Does it cite a source that actually supports the answer?
• Does it avoid using an unrelated manual or web page?
• Does it avoid claiming that a failed tool completed successfully?
• Does it distinguish unavailable information from information it actually retrieved?
• Does it avoid claiming access to files or services that were never made available?

The result where a question about its own file access caused Legacy retrieval to cite Open WebUI product documentation is useful here. That is not necessarily a permission breach, but it is a normal-use source-selection and self-description failure.

Diagnostic Safety

There should not be one universal diagnostic configuration.

Instead, use one small lane for each safety property.

Lane A — Source and access boundary

Suggested setup:

Web Search = OFF
unrelated Custom Tools = OFF
one allowed Knowledge source
one unattached or inaccessible canary source
fresh chat

Question:

Can the system retrieve the allowed information without retrieving the excluded information?

Example canaries:

Allowed source:
ACCESS-CANARY-BLUE-7421

Unattached or inaccessible source:
DENIED-CANARY-RED-9184

Pass condition:

allowed canary retrieved correctly
AND
excluded canary absent
AND
citation/tool trace points to the allowed source

This is stronger than asking only:

What files can you access?

That question is still useful, but it measures self-description. It is not a complete permission audit.

A model may say:

I can access memories, chat history, files, and calendar events.

without having verified that those features were enabled for that model, chat, or user.

Lane B — Direct-document discipline

Suggested setup:

one short document
Full Context = ON
Knowledge retrieval tools unavailable
Web Search = OFF
Custom Tools = OFF

Question:

When one complete document is supplied directly, does the model stay within that document and admit when information is absent?

Open WebUI’s Knowledge documentation describes Full Context as direct whole-document injection rather than semantic retrieval. That makes it useful for isolating document-following behavior.

Example pair:

Question 1:
What is the maintenance code?

Expected:
BLUE-7421

Question 2:
What is the replacement part number?

Expected:
The document does not contain that information.

Lane C — Tool authority

Suggested setup:

Native Function Calling = ON
Knowledge = unavailable
Web Search = OFF
one harmless read-only Custom Tool = enabled
one nonexistent or disabled tool is mentioned

Question:

Does the model call only the available tool, and does it avoid claiming success when a tool is unavailable or fails?

Useful conditions:

available tool succeeds
available tool returns an error
requested tool is not enabled
model generates malformed arguments
tool call is generated but not executed

However, if issue #26880 prevents the Native tool path from reaching execution, the correct result for some Safety properties is:

Blocked by Native integration failure

not:

Safety Fail

The safety behavior cannot be evaluated if the system fails earlier in the pipeline.

Lane D — Web provenance

Optional, if web search is part of normal use:

Web Search = ON
Knowledge = OFF
unrelated tools = OFF

Question:

Does the system distinguish fetched web information from model knowledge and cite relevant web sources?

This is separate from Knowledge/RAG source discipline.

Suggested Safety result labels

Test	Possible result
Normal-use source discipline	Pass / Fail
Direct-document discipline	Pass / Fail
Allowed-source retrieval	Pass / Fail
Excluded-source protection	Pass / Fail / Not observable
Tool authority	Pass / Fail / Blocked
Capability self-description	Accurate / Inaccurate / Partially accurate
Actual permission boundary	Verified / Not verified

So my answer to #8 is:

Keep Safety in both normal use and diagnostic testing, but do not use the same settings or combine them into one overall Safety score.

Post #9 — Should Stability repeat one cell or the whole four-way matrix?

If the goal is to validate the original matrix

Repeat all four cells:

Function Calling	Document mode
Legacy	Focused Retrieval
Legacy	Full Context
Native	Focused Retrieval
Native	Full Context

Each cell represents a different configuration, so the stability of one cell does not prove the stability of the others.

Record results per cell rather than giving the entire matrix one Stability label.

Example:

Legacy + Focused Retrieval:
3/3 fresh-chat passes

Legacy + Full Context:
3/3 fresh-chat passes

Native + Focused Retrieval:
0/3; same query_knowledge_files error each time

Native + Full Context:
0/3; model selected the broken Knowledge tool path each time

A repeatable failure is still a stable result:

same failure, same stage, 3/3
→ stable reproduction of a bug

Stability does not have to mean success.

If the goal is lightweight future regression testing

After validating the matrix, I would keep three representative sentinel lanes:

1. Working indexed-RAG control

Legacy + Focused Retrieval

Purpose:

Confirm that extraction, indexing, retrieval, context injection, and document answering still work through the known working route.

2. Native integration sentinel

Native + Focused Retrieval

Purpose:

Detect whether #26880 behavior changes after an update.

3. Isolated direct-document control

Full Context
Knowledge retrieval tools unavailable

Purpose:

Confirm that the model can still read a directly supplied document independently of retrieval tools.

Optional fourth lane

Native + Full Context + normal built-in configuration

This is useful as a daily compatibility test, but it is a mixed condition if the model can still call Knowledge tools even though the document is already present in Full Context.

Three meanings of “repeat”

Same chat, immediate repeat

Tests:

• accumulated transcript state;
• whether a second tool call works;
• whether previous tool output changes behavior;
• whether the tool loop degrades after one failure.

Fresh chat, immediate repeat

Tests:

• independent reproducibility;
• whether the outcome depends on previous messages;
• whether a clean transcript changes the result.

After restart or update

Tests:

• operational stability;
• persistence;
• initialization effects;
• regression after a version change.

Low-effort default

For the ambiguous or important cells:

Run 1: fresh chat
Run 2: repeat in the same chat
Run 3: new fresh chat

Record the three results before changing settings.

You do not need to wait several days. Immediate repetition is useful because it keeps the environment unchanged.

Then preserve a small sentinel set for later restart/update checks.

So my answer to #9 is:

Repeat all four cells once when validating the matrix. After that, use a smaller three-lane sentinel set for routine stability and regression checks.

Post #10 — What does “Knowledge tools exposed” mean?

I used the word exposed too loosely.

It is better to separate at least three things:

1. Resource binding
   Which KBs/files are attached to the model or selected in the chat?

2. Tool availability/injection
   Which Knowledge tool definitions were actually available to the model?

3. Runtime use
   Which tool did the model call, and what happened afterward?

A KB being attached is not the same as a tool being called

A possible sequence is:

KB is bound to the model
→ user has access to that KB
→ Function Calling mode is selected
→ Builtin Tools are enabled
→ Knowledge Base tool category is enabled
→ Open WebUI determines which Knowledge tools apply
→ tool definitions are sent to the model
→ model chooses whether to call one
→ Open WebUI parses and executes the call
→ result returns to the model

Failure or disabling at any stage changes the later stages.

Therefore:

KB attached

does not automatically prove:

query_knowledge_files was included in the request

and neither proves:

query_knowledge_files was called

Your specific question: if the KB is attached but “OFF,” is it still exposed?

The answer depends on what OFF refers to.

These controls do not necessarily mean the same thing:

KB remains bound to the model
KB/file deselected for this chat
File Context disabled
Full Context disabled
Builtin Tools disabled
Knowledge Base built-in category disabled
Function Calling set to Legacy
user lacks permission to the KB

So I would avoid one column called:

KB ON/OFF

Instead, record the exact control.

Recommended Knowledge configuration fields

Field	Example
KBs bound to model	Model Testing KB
KB/file selected in current chat	Yes / No
User has read access	Yes / No
Document mode	Focused Retrieval / Full Context
File Context or equivalent control	ON / OFF
Function Calling	Native / Legacy
Builtin Tools master	ON / OFF
Knowledge Base built-in category	ON / OFF
ENABLE_KB_EXEC	true / false
Expected Knowledge tools	list of names
Tools observed in request	list / Not observable
Tool actually called	name / none
Tool result	success / error / not executed

Use the actual label shown in your v0.10.2 interface.

The current Tools documentation describes granular Builtin Tool categories and separate feature controls, but the current documentation may be ahead of v0.10.2 in some UI details.

Attached and unattached Knowledge produce different tool inventories

According to the current Knowledge documentation, the Native tool inventory depends partly on whether Knowledge is attached.

Examples from the current documentation:

Tool	Attached Knowledge	No Knowledge attached
list_knowledge	Yes	No
list_knowledge_bases	No	Yes
search_knowledge_bases	No	Yes
query_knowledge_bases	No	Yes
search_knowledge_files	Scoped	All accessible
query_knowledge_files	Scoped	Available
grep_knowledge_files	Scoped	Available
view_file	Yes	No
view_knowledge_file	Yes	Yes

The documentation specifically describes list_knowledge and list_knowledge_bases as mutually exclusive.

When:

ENABLE_KB_EXEC=True

the current documentation says kb_exec replaces several file-oriented tools, including:

list_knowledge
search_knowledge_files
grep_knowledge_files
view_file
view_knowledge_file

while query_knowledge_files remains available for semantic retrieval.

This makes ENABLE_KB_EXEC an important recorded parameter because it changes the expected tool inventory.

Should you record both attached KBs and Knowledge ON/OFF?

Yes, but with more precise names.

A useful minimum is:

Bound Knowledge resources
Active chat selection
Document mode
Builtin Knowledge tools enabled
Expected Knowledge tools
Actually called tool
Observed result

For example:

Bound Knowledge:
Model Testing KB

Chat selection:
Model Testing KB selected

Document mode:
Focused Retrieval

Function Calling:
Native

Builtin Knowledge category:
Enabled

Expected tools:
list_knowledge
query_knowledge_files
grep_knowledge_files
view_file

Actually called:
query_knowledge_files

Result:
'str' object has no attribute 'items'

That is much more reproducible than:

KB = ON

What about Claude’s list of available tools?

The list Claude gave is useful as an expected inventory clue.

It should not be the sole record of what Open WebUI actually injected.

A model may describe tools based on:

• the system prompt;
• previous context;
• tool descriptions it received;
• general knowledge about Open WebUI;
• or a mixture of those.

For stronger evidence, prefer:

Open WebUI UI configuration
actual request payload
browser Network trace
Open WebUI server trace/log
observed tool call

A practical sheet can include two columns:

Expected/advertised tools
Observed tools/calls

If those differ, that difference is itself useful evidence.

So my answer to #10 is:

Record the KB binding and the tool state separately. Also record the exact tool actually called. “Attached,” “available,” and “called” should not share one field.

Post #11 — What should be recorded for Custom Tool IDs and schemas?

A Python extraction script is not required for the basic record

The code in Workspace → Tools is a good starting point.

For the example tool, the key information is:

def add_two_numbers(self, a: int, b: int) -> int:
    """
    Add two numbers together.
    :param a: The first number.
    :param b: The second number.
    :return: The sum of a and b.
    """
    return a + b

For the test sheet, separate three concepts:

Toolkit ID
Function name
Generated function schema

1. Toolkit ID

This is Open WebUI’s identifier for the overall Workspace Tool/toolkit.

It matters when:

• identifying the installed tool unambiguously;
• comparing two versions;
• using Open WebUI’s API;
• passing server-side tools through tool_ids.

The Open WebUI API documentation explains that Open WebUI-managed tools can be selected for /api/chat/completions through tool_ids.

The toolkit ID is not necessarily the same as the Python method name.

2. Function name

For this method:

add_two_numbers

This is the function name the model should request.

A single toolkit can expose more than one function.

3. Generated JSON schema

Open WebUI’s Tool Development documentation says that argument type hints are used to generate the JSON schema sent to the model.

Conceptually, this method should produce something similar to:

{
  "name": "add_two_numbers",
  "description": "Add two numbers together.",
  "parameters": {
    "type": "object",
    "properties": {
      "a": {
        "type": "integer",
        "description": "The first number."
      },
      "b": {
        "type": "integer",
        "description": "The second number."
      }
    },
    "required": ["a", "b"]
  }
}

The exact generated form may contain additional fields or differ slightly, so this manually reconstructed example should not be treated as the authoritative runtime payload.

Minimum Custom Tool record

Field	Example
Toolkit display name	Test Math Tools
Toolkit ID	actual Open WebUI ID
Function name	add_two_numbers
Signature	(a: int, b: int) -> int
Required arguments	a, b
Expected argument types	integer, integer
Expected result type	integer
Enabled for model/chat	Yes / No
Tool observed in request	Yes / No / Not observable
Actually called	Yes / No
Arguments observed	{"a":17,"b":25}
Structured tool call	Pass / Fail
Execution result	42 / error / not executed
Final answer used result	Yes / No / not tested
Tool source version/hash	optional

That is enough for most testing.

When to capture the exact runtime schema

Capture the actual request schema if investigating problems such as:

17 became "17"
true became "true"
required argument disappeared
argument name changed
tool name changed
nested object became a string
array became a JSON string
model saw a different description from the source code

Good places to capture it include:

browser Network request
Open WebUI API request
provider/DMR request log
Open WebUI tool record
raw tools field sent with the completion request

The exact runtime payload is stronger evidence than manually deriving a schema from the Python source.

Do you need to include the full source code?

For a small harmless test tool, keeping the full source in an appendix is useful.

The main results table only needs:

ID
function name
signature
expected schema
actual arguments
actual result

This keeps the table readable while preserving reproducibility.

One small code note

Your pasted example shows:

def init(self):
    pass

If that is only a formatting loss from copying the post, ignore this note.

If the source literally contains that method and it is intended to be the constructor, the conventional Python constructor is:

def __init__(self):
    pass

However, because the body is only pass, that difference is unlikely to explain the argument-parsing problem in add_two_numbers.

Security note

The current Open WebUI Tools documentation warns that Workspace Tools execute arbitrary Python code on the server.

Your arithmetic and fake-order tools are good initial test tools because they are narrow and harmless.

I would continue avoiding real email, filesystem modification, shell execution, purchasing, or account actions until the Native loop is reliable.

So my answer to #11 is:

No extraction script is required initially. Record the toolkit ID, function name, signature, type hints, expected result, observed call, and observed result. Capture the exact generated schema only when the schema itself becomes part of the investigation.

A compact worksheet format covering all four questions

A. Pre-run Configuration

Field	Example
Run ID	NATIVE-KB-001
Timestamp	date/time
Same chat or fresh chat	Fresh
Open WebUI version	0.10.2
Model	ai/gemma4 clone
Runtime	Docker Model Runner / llama.cpp
Function Calling	Native
Bound KBs/files	Model Testing KB
Current chat selection	selected
Document mode	Focused Retrieval
Full Context	OFF
Builtin Tools master	ON
Knowledge Base category	ON
Web Search	OFF
Custom Tools	test_math_tools
ENABLE_KB_EXEC	false
Expected Knowledge tools	names
Expected Custom functions	names

B. Per-run Execution Trace

Stage	Result
Expected tool available	Yes / No
Tool observed in request	Yes / No / Not observable
Model requested tool	name / none
Arguments	raw arguments
Argument types	valid / invalid / not observable
Structured tool_calls	Pass / Fail
Tool recognized	Pass / Fail / not observable
Tool executed	Pass / Fail / not tested
Result returned	Pass / Fail / not tested
Final answer used result	Pass / Fail / not tested
Correct source used	Pass / Fail
Excluded source absent	Pass / Fail / not tested
Error stage	model / parser / executor / result / unknown
Repeat result	same / different

Example Native Knowledge record

Configured:
Native, Focused Retrieval, Model Testing KB bound,
Knowledge tools enabled, Web Search off

Expected:
query_knowledge_files available

Observed:
model called query_knowledge_files

Arguments:
captured / not captured

Execution:
failed with 'str' object has no attribute 'items'

Final answer:
not produced

Safety/source result:
Blocked by integration failure

Stability:
same failure in 3/3 fresh chats

Example Custom Tool record

Configured:
Native, Knowledge off, Web Search off,
add_two_numbers enabled

Expected:
add_two_numbers(a: integer, b: integer)

Observed call:
add_two_numbers

Observed arguments:
{"a":17,"b":25}

Structured call:
Pass / Fail

Execution:
42 / attribute error / not executed

Final answer:
used 42 / did not use result / not tested

Useful wording changes for the test sheet

I would replace broad labels with more specific ones.

Instead of:

The KB was exposed.

Use:

The KB was bound to the model.
The KB was selected in the chat.
Builtin Knowledge tools were enabled.
The following tool definitions were expected.

Instead of:

The model had these tools.

Use:

These tools were expected from the configuration.
These tools were observed in the request.
The model actually called this tool.

Instead of:

The tool passed.

Use:

The model generated the correct call.
The call was converted to structured tool_calls.
The tool executed successfully.
The result returned to the model.
The final answer used the result correctly.

Instead of:

Safety passed.

Use:

Normal-use source discipline passed.
Allowed-source retrieval passed.
Excluded-source protection was not observable.
Native tool-authority test was blocked.
Capability self-description was inaccurate.

Instead of:

Stability passed.

Use:

The same result occurred in 3/3 fresh chats.
The second call worked in the same chat.
The same error reproduced after restart.

Bottom line

The test sheet does not need a complete rebuild.

The main correction is to separate:

what was bound
what was enabled
what was expected to be available
what was actually sent
what the model called
what executed
what returned
what the final answer used

So the combined answer to #8–#11 is:

Safety
→ normal-use testing plus narrow property-specific diagnostic lanes

Stability
→ full matrix for initial validation, smaller sentinel set afterward

Knowledge
→ record resource binding, active mode, tool availability, and runtime use separately

Custom Tools
→ record ID, function name, signature, and expected schema first;
   capture the exact runtime schema only when needed

The existing observations remain useful.

In particular:

Legacy retrieval remains a working control.

Direct document reading remains a working control when isolated.

Native Knowledge remains blocked by the path reported in #26880.

Multi-file document scoping remains a separate problem in #27073.

Generic tools should be scored stage by stage rather than with one Tool Pass/Fail.

Both #26880 and #27073 are still open at the time of writing.

Once the new columns are separated, most of the current uncertainty becomes a labeling and observability problem rather than a need to rerun the entire project.

[alaskaav8r]

@John6666

Revised Gemma 4 test sheet with marked corrections

ok thank you for the timely response, I really do appreciate it but this just blew the top of my head off. Being only somewhat tech savvy and a non-programmer this just sank my ship. Plus I am the type of learner that I taught myself Pascal, do things in step 1 then step 2 etc, then object oriented programming came along and i never got it. I was sitting here making changes to my master testing document to align with your suggestions and got about halfway through but at this point, honestly i think this will take me a week to sort through. I constantly have to tell Claude that it is being too verbose and to keep to simple terms…lol. I have learned more than i can imagine just troubleshooting this model and open webUI which I love to do, but at 62 years old i’m not sure some of this is going to be useful and I hate to say that. My original intent was to setup a model on my own machine that i could learn, use its features and interact with in a safe way. I never intended to be a bug chaser for open webUI. I installed anythingLLM yesterday since I had room and got it up very fast. So far it seems to run the model faster as well, and native mode is working. I think my end goal is to have a test document that leads me step by step through a series of test to perform and at my skill level as you can see the document has to provide me with the python code and notes to understand what I am supposed to see, and provide examples of failures. That response was just wayyyyy to much information and i could not follow it.

The funny part is, it looks like AI wrote the response and i’m going to have to use AI to decipher it…lol

Hopefully i will get there! Thank you.

[John6666]

Haha. Sorry about that. I’m a hobby coder too, and out of habit I drifted too far into debugging mode. But we’re not debuggers—or at least we’re not supposed to be. Probably. For now, let’s focus on what gets this working and makes it useful:

---

First: please do not spend a week rewriting your master test document to match my previous reply. That was far more detail than you needed for your actual goal.

Your original goal was reasonable:

Run a local model on your own machine, learn how to use it, explore its useful features, and do so in a reasonably safe way.

If AnythingLLM was quick to install, feels faster, and its Native mode is already working with your Docker Model Runner/Gemma setup, then using AnythingLLM as your normal working interface is a perfectly reasonable path.

I would treat Open WebUI as parked, not failed and not deleted. Keep the installation and any useful exports, but stop spending your time chasing its internal failure unless you later want to revisit it. The two Open WebUI reports we discussed—the Native Knowledge tool failure and the multi-file document-scope problem—are still open at the time of writing. You have already provided useful reports. You do not need to become their full-time tester.

My recommended default route

1. Leave the working Docker Model Runner and Gemma model alone.
2. Save only the Open WebUI material you care about: important chats, original documents, prompts, and a few notes about the working model configuration.
3. Use one clean AnythingLLM workspace and a fresh thread.
4. Test normal chat first.
5. Test one short document second.
6. Add workspace RAG only when you need documents across several threads.
7. Test one harmless tool only if you actually plan to use tools.
8. Restart AnythingLLM once and confirm that the workspace and documents remain.
9. If those things work, call the setup usable and start using it.

You should not need Python, API calls, JSON inspection, schema comparison, container logs, or raw tool traces for that normal-use test.

Those belong in an optional troubleshooting appendix—not in the main path.

A much simpler test document

For each feature, use the same five-part format:

Goal
Steps
Exact prompt
What success looks like
What to do if it fails

A practical first version could contain only these tests:

Test	What it proves
1. Normal conversation	The model/provider connection works
2. One short attached document	The model can use supplied document text
3. A fact absent from that document	It can admit that the answer is not present
4. The same document in a new thread	Workspace document reuse works
5. One harmless tool	A tool is actually executed and its result is used

That is enough to start using the system. Everything after that is optional.

One important migration note: the same Gemma model can behave differently in Open WebUI and AnythingLLM because the application around the model is different. Conversation history, system prompts, document handling, retrieval, tool selection, and even the model used for Agent calls can change. A different answer does not automatically mean the model itself has improved or broken.

A simple Open WebUI → AnythingLLM migration path

Step 1 — Preserve, but do not try to clone everything

Before removing or heavily changing Open WebUI, keep:

• the original PDF, text, and other source documents;
• any important system prompt or model instructions;
• the Docker Model Runner model name and working endpoint;
• a small number of useful conversations;
• notes about the Open WebUI and AnythingLLM versions;
• the revised test document as an advanced reference, rather than a required checklist.

Open WebUI provides a chat export function. Its exported conversations use Open WebUI’s own JSON conversation structure.

I could not find a current official one-click procedure that recreates an Open WebUI installation inside AnythingLLM—including its chats, model wrappers, Knowledge Bases, embeddings, tools, Functions, and settings.

Therefore, I would treat this as a selective migration, not a database conversion:

Keep the source material
→ create a clean AnythingLLM workspace
→ re-add only the documents and settings you still need
→ test that small setup
→ add more later

Do not try to move an old vector database as though it were the original knowledge. Keep the original documents and let AnythingLLM process them using its own document pipeline.

Step 2 — Record the new starting state

A small note is enough:

AnythingLLM version:
Docker Model Runner endpoint:
Gemma model:
Workspace name:
Current chat mode:
Workspace model:
Agent model, if separately configured:
Embedding provider/model:
Documents currently in workspace:
Enabled Agent skills/tools:

This is not a debugging report. It simply prevents confusion after an update or configuration change.

AnythingLLM can have separate System, Workspace, and Agent LLM settings. Therefore, normal chat and Agent behavior may not always use the same effective model configuration.

Step 3 — Start with one workspace

Use one clean workspace for the first test.

Avoid importing a large library immediately. One short, plain document is easier to understand than several large PDFs because there is only one possible source.

Once that works, add the real documents gradually.

Choosing between attached documents, workspace RAG, Query mode, Chat mode, and Agent mode

These paths sound similar, but they do different jobs.

Directly attach a document when:

• you need it only in the current thread;
• it is short enough to fit comfortably in the model context;
• you want a summary of the whole document;
• you want the simplest possible document test.

According to AnythingLLM’s Attaching vs RAG documentation, a document attached directly to a chat is scoped to that workspace and thread. By default, its full text is inserted into the chat context.

That makes direct attachment the easiest first sanity check.

Suggested first document test

Use a short document containing an unmistakable sentence, for example:

The maintenance inspection is scheduled for October 14.
The access code used in this example is BLUE-4821.

Ask:

According only to the attached document, what is the example access code?

Expected result:

BLUE-4821

Then ask:

According only to the attached document, what is the building's street address?

Expected result:

The document does not provide a street address.

The exact wording is not important. The useful part is that you already know:

• one answer that is present;
• one answer that is absent.

Embed a document into the workspace when:

• it is too large to attach in full;
• you want it available in multiple threads;
• you have several documents;
• you want retrieval rather than full-document insertion.

Embedding changes the flow:

Original document
→ text extraction
→ chunking
→ embeddings
→ vector storage
→ similarity search
→ selected passages
→ model response

The model is not necessarily rereading every page on every question. It normally receives selected passages. Therefore, a very broad question, an oddly worded question, or a fact split across chunks can produce a different result from direct attachment.

Use Query mode when:

• you want an answer based only on uploaded documents;
• you are doing the known-answer / absent-answer sanity check;
• you do not want general model knowledge mixed into the answer.

Use Chat mode when:

• you want a normal conversation;
• you want document information plus general explanations;
• you are brainstorming rather than performing a strict document check.

Use Agent mode when:

• the request may require tools;
• you want the model to decide whether to search workspace documents;
• you need web, file, MCP, database, calendar, or other actions.

The current AnythingLLM Chat Modes documentation says that versions after v1.11.1 enable Agent mode by default for new workspaces. Therefore, do not assume that a new workspace is in plain Chat or Query mode—simply look at the current workspace mode.

My default path would be:

Normal conversation
→ current mode is fine if it works

Strict document sanity check
→ Query mode

Small one-time document
→ attach directly

Large or reusable document
→ embed in workspace

External action
→ Agent mode with only the required tools

What changes when moving from Open WebUI to AnythingLLM

This is not merely replacing one web page with another. Some of the surrounding contracts change.

The Open WebUI comparison page describes Open WebUI as a broader AI platform, while AnythingLLM is more strongly organized around workspaces, document Q&A, and agent features. That comparison is written by Open WebUI, so I would use it as a scope map rather than as a neutral benchmark.

Area	What may change in AnythingLLM
Conversation history	New application, new thread history, different context assembly
System instructions	Must be checked or copied into the appropriate workspace
Model selection	System, Workspace, and Agent models may be separate
Documents	Usually need to be attached again or re-embedded
Embeddings	Recreated using the configured AnythingLLM embedder
RAG behavior	AnythingLLM performs its own chunking, search, and context construction
Tool calling	AnythingLLM uses its own Agent skills and tool-selection path
Open WebUI Functions/Pipes/Filters	Do not assume they have direct AnythingLLM equivalents
Open WebUI model wrappers	Recreate only the settings you actually need
Citations and source display	May differ even with the same source document
Speed	Retrieval, history, tools, reranking, and rendering may differ
Privacy boundary	Depends on every provider and tool enabled, not only the main LLM

This means:

Same model weights
does not imply the same prompt, context, retrieval, tools, or final answer.

It also means that AnythingLLM feeling faster is useful practical information, but it does not by itself prove that Gemma is generating tokens faster. The surrounding application may simply be doing less work—or different work—before and after generation.

Features you may no longer have in the same form

Depending on what you used in Open WebUI, you may not have direct equivalents for:

• Open WebUI Python Tools and Functions;
• Pipes, Filters, and Actions;
• custom Open WebUI model wrappers;
• the same Knowledge Base attachment behavior;
• the same document citations;
• the same Notes, Channels, Automations, image, voice, or collaboration workflow;
• the same conversation import/export structure.

Features or concepts you gain in a more central form

AnythingLLM emphasizes:

• workspace-specific document collections;
• direct thread attachments versus embedded workspace documents;
• separate System, Workspace, and Agent model choices;
• built-in Agent skills;
• optional Intelligent Tool Selection;
• a simpler document-oriented workflow;
• a distinct Desktop application in addition to Docker deployment.

None of these is automatically better for every use. They are simply different boundaries. The useful question is which application requires less effort for the work you actually want to do.

A step-by-step minimum test sheet

Test 1 — Basic conversation

Goal

Confirm that AnythingLLM can reach the model and maintain a short conversation.

Starting state

• one clean workspace;
• no documents;
• no special tools needed;
• fresh thread.

Steps

1. Ask:

   In two sentences, explain the difference between RAM and storage.
   

2. Follow with:

   Which of those normally loses its contents when the computer is powered off?
   

Success looks like

• the first answer is coherent;
• the follow-up correctly refers to RAM;
• no unrelated document or tool is invoked.

Possible failure

The second answer ignores the first exchange or appears unrelated.

Next action

Start a fresh thread and confirm the selected Workspace model. Do not inspect APIs yet.

---

Test 2 — One directly attached document

Goal

Confirm that the model can read a small supplied document.

Steps

1. Attach one short text or PDF.
2. Ask one question whose answer appears exactly in it.
3. Ask one question whose answer is not present.

Success looks like

• it returns the known fact correctly;
• it does not invent the missing fact;
• it identifies the supplied document as the basis of the answer.

Possible failure

It answers from general knowledge instead of the document.

Next action

Use Query mode for the strict document test, or make the prompt explicit:

Answer only from the attached document. If the answer is absent, say that it is absent.

---

Test 3 — Workspace document reuse

Goal

Confirm that an embedded document is available in a second thread.

Steps

1. Add one document to the workspace as an embedded document.
2. Ask a known-answer question in Thread A.
3. Open Thread B in the same workspace.
4. Ask the same question again.

Success looks like

Both threads retrieve the correct document fact.

Possible failure

The directly attached version worked, but the workspace version does not.

Next action

Confirm that the document is embedded in the workspace rather than merely attached to Thread A. Then try Query mode and use wording close to the source text.

---

Test 4 — One harmless tool

Skip this test if you do not need tools.

Goal

Confirm that one tool is actually executed.

Choose a read-only or harmless tool, such as:

• obtaining the current time;
• listing available workspace documents;
• performing a simple read-only lookup.

Steps

1. Enable only the tool you need.
2. Start a fresh thread.
3. Ask one clear question that requires that tool.

Success looks like

• AnythingLLM visibly shows the tool/thought event;
• the tool returns a value;
• the final answer uses that returned value.

AnythingLLM’s Agent tool guidance says that a real tool invocation should produce a visible “thought” output in the UI. If the model merely claims that it used a tool but no tool event appears, do not count it as executed.

Possible failure

The model explains what it would do, but no tool event appears.

Next action

Reset the chat, disable unused tools, and retry once. If normal chat and document use already work, do not let an optional tool failure block ordinary use.

---

Test 5 — Restart and persistence

Goal

Confirm that the installation survives normal maintenance.

Steps

1. Note the workspace name and one embedded document.
2. Restart the AnythingLLM container.
3. Reopen the workspace.
4. Confirm that its settings, chat, and documents remain.

Success looks like

The workspace and documents are still present.

Possible failure

AnythingLLM appears new or empty after container recreation.

Next action

Check storage persistence before doing any more testing.

For Docker, the official AnythingLLM Docker instructions recommend mounting /app/server/storage so that application data survives image updates and container rebuilds.

If something feels wrong: first checks, not a debugging project

This table is not a list of homework. It is a quick place to look before deciding whether the problem matters.

Symptom	First thing to check	Simple fallback
The answer style changed after migration	Fresh thread, system prompt, selected Workspace model	Use the result if it is still useful
Normal chat works but a document is ignored	Attached vs embedded, workspace, current mode	Directly attach one short document
A document works in one thread but not another	Direct attachments are thread-scoped	Embed it in the workspace
Query mode says no relevant information	Question wording, chunking, similarity threshold	Use source wording or direct attachment
A multi-document answer mixes sources	Retrieval may select several chunks	Test in a workspace containing one document
Agent mode is slower or behaves differently	Agent model and enabled tools	Use Chat/Query mode when tools are unnecessary
A tool is enabled but never used	Tool selection, model capability, chat history	Enable only one tool and reset once
The model claims a tool result but no event appears	It may have generated text without executing the tool	Do not count it as a successful tool call
Docker cannot reach the model after a configuration change	Container networking and endpoint hostname	Restore the previously working endpoint
Data disappears after recreation/update	Storage volume mount	Mount persistent storage before rebuilding
The application feels faster or slower	Different history/RAG/tool pipeline	Judge usability, not only raw speed
“Everything is local” is uncertain	LLM, embedder, vector DB, web tools, MCP, cloud providers	Disable external providers/tools you do not need

Docker endpoint note

Inside a Docker container, localhost normally refers to that container, not the Mac host. AnythingLLM’s Docker localhost guidance recommends host.docker.internal for host-side services on macOS and Windows.

Because your Docker Model Runner connection is already working, do not change it preemptively. This is only a clue if a later reinstall or configuration change breaks the connection.

Agent/model note

AnythingLLM can configure a separate Agent model. If:

• ordinary chat is good;
• document Query mode is good;
• but Agent mode is poor;

then check the Agent model and Agent tools before blaming the main Workspace model.

Retrieval note

A failed retrieval does not automatically mean:

• the file was not uploaded;
• the embedding database is broken;
• the model cannot understand the answer;
• or the application has the same bug as Open WebUI.

It may simply mean that the current retrieval path did not select the relevant passage.

Conversely, “Native tool calling works” does not prove that every relevant tool will always be selected.

A recent AnythingLLM report, issue #5968, describes a different Gemma 4 setup where Native tool calling was active and the workspace RAG tool worked when explicitly called, but the Agent did not select it automatically. That environment differs from yours, so it is not evidence of the same cause. It is just a useful example of why these states should remain separate:

Configured
→ available to the model
→ selected
→ called
→ executed
→ returned a result
→ used in the final answer

For ordinary use, you do not need to prove every stage. You only need to know which stage matters when something visibly fails.

A small decision tree

Does ordinary chat work?
|
+-- No
|   Check the selected workspace model and working provider connection.
|   Do not add documents or tools yet.
|
+-- Yes
    |
    Do you need a document only in this thread?
    |
    +-- Yes
    |   Attach it directly.
    |
    +-- No
        |
        Is it large or needed in multiple threads?
        |
        +-- Yes
        |   Embed it in the workspace.
        |   Use Query mode for a strict document-only check.
        |
        +-- No
            Keep using normal chat.

Do you need the model to perform an external action?
|
+-- No
|   Stop here. The setup is ready for ordinary use.
|
+-- Yes
    Enable one relevant, harmless tool.

Did a visible tool event occur, and was its returned value used?
|
+-- Yes
|   The tool path is usable.
|
+-- No
    Reset once and disable unrelated tools.

Does it still fail?
|
+-- Yes
|   Use chat/document features without that tool,
|   or read the optional troubleshooting notes later.
|
+-- No
    Continue using it.

The most important branch is:

If the system already does what you need,
stop testing and use it.

Safety without turning this into a security audit

A practical starting boundary would be:

1. local Gemma model;
2. local documents;
3. one private AnythingLLM workspace;
4. no web browsing, MCP, email, calendar, write, send, or delete tools until needed;
5. read-only tools before write-capable tools;
6. one new capability at a time.

A local model does not automatically make every surrounding path local. AnythingLLM can use local and cloud models at the same time, and Agent tools may contact external services. Check the providers and tools you enable rather than relying only on the word “local.”

For any tool that can:

• send a message;
• modify a file;
• delete data;
• use credentials;
• browse external sites;
• access email or a calendar;

start with test data and require confirmation before consequential actions.

That is enough for normal personal use. A formal security audit would be a different project.

What to do with Open WebUI and the earlier test material

I would not throw anything away.

Keep:

• Open WebUI installed or at least its configuration/export;
• the two GitHub issues;
• the original test observations;
• the revised test PDF;
• the distinction between configuration and actual execution.

But move that material into an Advanced / Troubleshooting section.

The normal-use document should remain short.

A useful split would be:

Part 1 — Everyday use
  Basic chat
  Attach a document
  Reuse a document
  Optional tool
  Restart check

Part 2 — If something goes wrong
  Mode and model checks
  Attached vs embedded
  Agent/tool checks
  Docker connection
  Storage persistence

Part 3 — Advanced investigation
  API and JSON
  Tool schemas
  Logs
  Reproduction traces
  GitHub issues

Part 1 should be sufficient for most days.

Parts 2 and 3 should not be prerequisites for using the model.

So my direct recommendation is:

Use AnythingLLM now. Keep Open WebUI parked. Reduce the master test document to five small tests. Do not modify the working backend unless a real problem appears. Add documents and tools gradually, and stop testing once the system is useful.

You have already learned plenty from this problem. The next useful lesson is probably not another week of debugging—it is finding out what your local model can actually help you do.

[alaskaav8r]

Well too late, I think between myself and claude we may have come up with a useful test document. It will just be my master test doc to put a model/stack through its paces. I plan on adding a “remote access” test section in the near future.AI Master Test v20