ctogaurav/GLM_OCR-GGUF

GLM-OCR v4.1 — GGUF (for LM Studio / Ollama / llama.cpp)

GGUF conversion of ctogaurav/GLM_OCR v4.1 — the LoRA adapter merged into zai-org/GLM-OCR and converted for local, single-file inference. Verified working — tested end-to-end on GPU, output checked against the model's own validated ground truth (see below), not just "conversion succeeded."

Files

Both versions are available. Each needs two files — the language model and its vision projector (mmproj). Download both files of whichever version you want, into the same folder.

Version	Files	Size
v4.1	v4.1/GLM-OCR-v4.1-Q8_0.gguf + v4.1/mmproj-GLM-OCR-v4.1-Q8_0.gguf	~683 MB + ~485 MB
v3.1	v3.1/GLM-OCR-v3.1-Q8_0.gguf + v3.1/mmproj-GLM-OCR-v3.1-Q8_0.gguf	~683 MB + ~485 MB

Which to pick: v4.1 has lower character error rate and better content metrics; v3.1 compiles to a valid PDF more often (88.9% vs 82.4%). See the main model card for the full comparison.

LM Studio detects the mmproj file automatically when it's named mmproj-* alongside the main model. With llama.cpp directly:

llama-mtmd-cli -m GLM-OCR-v4.1-Q8_0.gguf --mmproj mmproj-GLM-OCR-v4.1-Q8_0.gguf --image page.png \
  -p "OCR this handwritten math page. Convert ONLY the handwritten mathematical content into a
      complete, compilable LaTeX document. Ignore printed text, student info, page numbers,
      cancelled work and rough work. Output only LaTeX." \
  -n 2048 --temp 0 --repeat-penalty 1.0 -c 8192

⚠️ Required settings — read this if output repeats forever

The most common failure is the model looping the same line endlessly instead of finishing. This is almost always a configuration mismatch, not a broken model. Required settings:

Setting	Value	Why
Context length	≥ 8192	The image alone consumes ~1536 tokens. At 2048 context the image fails to decode outright; too-small contexts are a prime cause of degenerate output. This is the #1 thing to check.
Temperature	0	The model was fine-tuned and benchmarked with greedy decoding.
Repeat penalty	1.0 (off)	Matches training/benchmark conditions.
Max output tokens	2048	Matches the benchmark cap.
System prompt	empty	The fine-tune never saw a system prompt. LM Studio injects "You are a helpful assistant" by default — clear it.
Prompt	use the exact prompt above, verbatim	The model was fine-tuned on this specific instruction; paraphrasing degrades it.

⚠️ Resize your images first (the #1 cause of runaway repetition)

These adapters were fine-tuned at 1536 image tokens ≈ 1.2 megapixels. A full-resolution scan or phone photo is typically 8–14 MP — around 10× more image tokens than the model ever saw in training. That is far enough off-distribution that the model degenerates into repeating the same line until it hits the token cap.

llama-mtmd-cli downscales to the budget automatically. LM Studio does not. A 14 MP photo there produces ~17,800 prompt tokens, versus ~1,650 when sized correctly.

Resize to roughly 1.2 MP (about 920×1300 for a portrait page) before uploading. A helper script is in the code repo:

python tools/resize_for_ocr.py page.jpg          # -> page_ocr.png
python tools/resize_for_ocr.py scans/ -o ready/  # whole folder

Verified: the same page that loops at full resolution transcribes completely and terminates correctly with \end{document} once resized to 1.2 MP.

Using Ollama? There's a second option: tools/ocr_proxy.py resizes images automatically, transparently, for every request — no manual step per image.

pip install flask requests pillow
python tools/ocr_proxy.py                                 # listens on :11500
$env:OLLAMA_HOST = "http://localhost:11500"                # PowerShell
ollama run glm-ocr-v4.1 "...prompt... unresized_page.jpg"  # works, gets resized in transit

Verified end-to-end: an 8.7 MP image that fails outright on context size, sent through the proxy, comes back as clean, correctly-terminated LaTeX with zero manual resizing.

This only covers Ollama (its CLI, or any app you can point at a custom OLLAMA_HOST/API URL) or LM Studio's separate "Local Server" mode, if you use that instead of its chat window. It cannot help LM Studio's built-in chat window — that talks directly to LM Studio's internal engine with no configurable network endpoint, so nothing external can intercept it. For that interface, resize manually with resize_for_ocr.py first — there's no way around it.

Honest caveat: even correctly configured, the model does not succeed on every page — its measured PDF compile rate is 82.4% (v4.1) / 88.9% (v3.1) on held-out data. Long, repetitive pages (e.g. multi-step matrix row reductions where consecutive steps look nearly identical) are the hardest case. That ~11–18% failure rate is a property of the model, documented in the project's own benchmark — not a defect introduced by this GGUF conversion.

A real bug we hit and fixed (useful if you're converting this architecture yourself)

Naively running convert_hf_to_gguf.py on this checkpoint fails to load with missing tensor 'blk.16.attn_norm.weight'. Root cause: GlmOcrForConditionalGeneration's conversion code sets block_count = num_hidden_layers + num_nextn_predict_layers (16 + 1 = 17), reserving a 17th block for a NextN/multi-token-prediction head — but the publicly released zai-org/GLM-OCR checkpoint (loaded via AutoModelForImageTextToText) doesn't actually ship those NextN weights. The official ggml-org/GLM-OCR-GGUF quant does include them (as unused tensors — llama.cpp's own loader logs unused tensor blk.16... ignoring for them), presumably converted from a checkpoint variant that has them.

Fix: patch conversion/glm.py's GlmOCRModel.__init__ to drop the + num_nextn_predict_layers term, so block_count matches what's actually in the checkpoint (16). Since NextN is confirmed unused at inference time regardless, this has no effect on output quality — verified by direct comparison against ground truth (below).

Verification, not just "it ran"

Tested against ctogaurav/GLM_OCR's own samples/ (2 pages with known, pdflatex-validated ground truth). Output matched the validated transcription almost verbatim — this confirms the merge + conversion + quantization pipeline preserves the fine-tuned model's actual behavior, not just that the file loads.

Additionally stress-tested on a multi-step matrix row-reduction page (the hardest known case for this model — consecutive near-identical matrices are what typically triggers runaway repetition). With the settings above, v4.1 transcribed it fully and terminated correctly with \end{document}. Both versions were verified this way before upload.

License

MIT, same as the base model and the source adapter — free to use, redistribute, and modify, including commercially.

Links

• Source adapter (PEFT, not merged): ctogaurav/GLM_OCR
• Code / full pipeline: github.com/realgauravvyas/ocr2tex
• Base model: zai-org/GLM-OCR