There are a few fully multimodal / omni-style large models, but if the more general goal is “I want my OSS local chat/RAG setup to call T2I/T2V”, I would usually make the image/video models separate local server processes and connect them as a pipeline. The execution cost, debugging cost, and replacement cost are usually lower that way. Existing frameworks already cover a lot of this:
Short answer
If you mean one offline Claude/Sonnet-like model or one magic USB app that also does high-quality T2I/T2V, then yes, I think that is probably pushing it.
If you mean a local pipeline, it becomes much more realistic:
chat / RAG / agent / workflow UI
→ tool call, HTTP request, CLI call, plugin, skill, or MCP
→ local image/video generation backend
→ output image/video file
So I would not start by trying to make the chat model itself render images or videos. I would start by making the media generator a small callable local backend.
The chat/RAG side could be LM Studio, Open WebUI, LibreChat, AnythingLLM, Dify, n8n, LangChain, LlamaIndex, Haystack, smolagents, a Python CLI, or something custom. The important part is not the UI. The important part is that the image/video generator has a narrow local interface.
The common pattern
A lot of existing tools already point in the same direction:
These are not the same stack, and I would not treat any one of them as the universal answer. But they show the general architecture:
LLM / RAG / agent layer
→ small tool interface
→ media-generation backend
→ file path, URL, or artifact
That is why I would frame this as a component-boundary problem rather than as a “find one bigger model” problem.
I would separate three questions
I would split this into three separate questions:
- How do I learn T2I/T2V itself?
- How do I run T2I/T2V locally as a callable process?
- How do I let a chat/RAG/agent layer call that process?
Those are related, but mixing them too early makes debugging much harder.
First practical target
Before involving RAG, agents, or MCP, I would first make this boring test pass:
prompt in
→ local process
→ output image/video file path out
That one test tells you whether the generation backend is real, local, reproducible, and scriptable.
More detail: learning T2I/T2V vs making it callable
Learning image/video generation is about models, prompts, workflows, LoRAs, ControlNet-like flows, img2img, inpaint, I2V/T2V, VRAM, speed, quality, and output handling.
Making image/video generation callable is a different problem. It is about having a stable contract:
input schema
→ backend
→ job status
→ artifact path
→ error handling
For learning the generation side:
For integration, I would not expose the whole generator to the LLM on day one. I would expose a narrow, boring interface first:
workflow_name or preset
prompt
negative_prompt, optional
seed, optional
width / height within bounds
maybe input_image path for img2img or I2V
return output path
I would avoid arbitrary workflow JSON, arbitrary shell commands, or arbitrary filesystem access as version 1.
ComfyUI path
In this context, I would not treat ComfyUI as “just a GUI”. I would treat it as a workflow-backed local inference server:
build workflow visually
→ export/use API form
→ call workflow from another process
→ return generated file
A practical first milestone:
- Build one known-good workflow manually in ComfyUI.
- Confirm it works locally.
- Export/use the API format.
- Expose only a few stable inputs.
- Call the workflow from a small Python script.
- Return an output file path.
- Only then connect it to chat/RAG/agent tooling.
The useful starting docs are:
The hard part is often not “can I send an HTTP request?” The harder part is “which inputs of this workflow are stable and safe to expose?”
More detail: ComfyUI backend notes, examples, and caveats
Useful related examples:
For ComfyUI especially, I would not make “the LLM invents arbitrary graphs” the first milestone.
There is research around generating or improving ComfyUI workflows, which is interesting, but it also suggests that this is non-trivial. See, for example:
For a practical local assistant, I would freeze one known-good workflow and expose only controlled inputs such as prompt, seed, size, preset, or input image path.
Diffusers path
If you are more comfortable in Python, Diffusers is the more natural route.
The basic shape is:
Python Diffusers pipeline
→ local FastAPI endpoint or CLI wrapper
→ output file path
Diffusers has an official guide for using a pipeline as a server inference engine: Create a server.
For images, this can be fairly straightforward:
/generate-image
input: prompt, seed, size, model/preset
output: image path
For video, I would treat it as a longer job:
/generate-video
input: prompt or input_image, seed, size, frames, preset
output: job_id first
then: progress/logs
finally: video path
That job-style design matters because video generation is usually slower, larger, and more failure-prone than single-image generation.
More detail: Diffusers/API-server examples
Useful links:
I would treat community projects as examples to inspect, not as proof that every model/workflow will work for your exact offline setup.
AUTOMATIC1111 / Forge path
If someone already uses the Stable Diffusion WebUI ecosystem, then AUTOMATIC1111 or Forge-style APIs are another practical path.
Useful links:
This path is useful when you want something like:
chat/agent tool
→ AUTOMATIC1111 API
→ txt2img/img2img
→ output image
It may be less ideal if your main target is complex video workflows, but it is a useful example of the same general architecture.
Where MCP fits
MCP is useful, but I would put it one layer above the renderer.
In other words:
MCP is not the image/video model.
MCP is a tool boundary that can call the image/video backend.
A reasonable path would be:
1. Make ComfyUI / Diffusers / A1111 callable locally.
2. Wrap it in a narrow HTTP or CLI interface.
3. Test it without an agent.
4. Then expose that interface as an MCP tool if you want multiple MCP-compatible clients to call it.
If a simple local HTTP API is enough, start there. MCP is useful once you need a standard tool boundary, tool discovery, or integration across several clients.
More detail: MCP / tool / agent integration links
Useful references:
The practical advice is: make the local backend boring and testable first. Then choose whether the wrapper should be direct HTTP, OpenAPI, MCP, a framework-specific tool, or just a CLI call.
Add video after image generation works
T2V/I2V uses the same architecture, but I would be more conservative with it.
Ready-made chat integrations are usually more mature for image generation than for video generation. For video, I would first prove the workflow locally, then wrap it as a job-style API with queue/progress/output-path handling.
A realistic order:
1. T2I: one local image from one prompt
2. img2img / inpaint / control workflow
3. callable local API
4. chat/RAG/tool integration
5. I2V/T2V after the image path is stable
Why video later?
- more VRAM,
- longer jobs,
- more output data,
- queue/progress handling matters,
- timeouts matter,
- FFmpeg or video encoding may matter,
- failure recovery matters,
- output file management matters.
More detail: video model/workflow examples
Examples worth knowing about, not necessarily first-step recommendations:
These show that OSS/local video generation is real, but they do not remove the engineering work around job duration, VRAM, queueing, output handling, and offline packaging.
Offline / travel checklist
For the travel/offline part, I would not trust “it worked once while online” as proof. I would do a real network-disabled test before depending on it.
Checklist:
- model weights are already downloaded,
- gated model access is already resolved,
- Python packages are installed,
- ComfyUI custom nodes/extensions are installed,
- FFmpeg/video tools are installed if needed,
- no hidden dependency on HF Inference API, Replicate, fal, Comfy Cloud, OpenAI, etc.,
- cache directories are on the disk you carry,
- output paths are predictable,
- logs do not leak sensitive prompts/images,
- the target machine has the required GPU driver / CUDA / ROCm / PyTorch stack,
- the app still runs after disconnecting the network completely.
USB/SSD can carry models, caches, environments, workflows, and scripts. It cannot magically carry the target machine’s GPU driver, VRAM, CUDA/ROCm compatibility, or OS-level dependency state.
More detail: related but not identical cases
These are not the same issue, but they are useful nearby examples:
The examples differ, but the recurring pattern is the same:
do not put everything inside one model;
make the generator a callable backend.
My suggested first build
If I were trying to make this practical, I would use this order:
Step 1:
Pick one backend:
ComfyUI, Diffusers, or AUTOMATIC1111/Forge.
Step 2:
Generate one image locally.
No RAG, no agent, no MCP yet.
Step 3:
Call the generator from a script.
prompt → backend → output path.
Step 4:
Freeze one known-good workflow/preset.
Expose only a narrow input schema.
Step 5:
Connect that script/API to the chat/RAG layer.
Step 6:
Only then add MCP/OpenAPI/tool metadata if useful.
Step 7:
Only then add video.
Treat video as a longer job with queue/progress/output handling.
That is less glamorous than “one fully multimodal offline assistant”, but it is easier to debug, easier to replace piece by piece, and closer to what existing OSS tools already support.
Installation and hardware compatibility.
A software / app that lets you install and use in 1 click, for everyone.
