tritueviet/sow-translate-1.5b-onnx

SoW Translate 1.5B — ONNX

A 1.5B-parameter multilingual translation model exported to ONNX, packaged to run in the browser with onnxruntime-web (WebGPU) or in Node with onnxruntime-node. It powers on-device subtitle translation in SubForge AI.

The model is a recurrent (RNN) language model: it carries a fixed-size state instead of a KV cache, so memory does not grow with context length and a conversation prefix can be snapshotted and reused — which is what makes per-line subtitle translation cheap here.

Variant	Files	Size	Notes
q4f16	q4f16/model.onnx + .onnx.data	~1.1 GB	4-bit block-quantized (MatMulNBits, block 32), fp16 activations. Recommended for browsers.
fp16	fp16/model.onnx + .onnx.data	~2.9 GB	Dense fp16. Reference precision.

Both were exported with chunk_size = 8 (see below). The repository also ships the tokenizer vocabulary (vocab.bin, 65,536 entries) in a compact binary form.

Graph contract

The graph has no control flow: the token loop is unrolled, so every operator is supported by the onnxruntime-web WebGPU backend. It consumes a fixed-size, left-padded chunk of tokens and carries the recurrent state through its inputs and outputs, so a single static graph serves both prefill and single-token decoding.

Inputs:

Name	Type	Shape
tokens	int64	[8]
mask	float32	[8] — 1 for a real token, 0 for left padding
state_att	float32	[24, 2048]
state_wkv	float32	[24, 32, 64, 64]
state_ffn	float32	[24, 2048]

Outputs: logits [65536] (for the last position only), plus state_att_out, state_wkv_out, state_ffn_out with the shapes above.

Padded steps are neutralised inside the graph (decay forced to 1, kk/v zeroed, token-shift reads the incoming state), so the result is bit-for-bit what you would get by feeding the real tokens alone. Start from all-zero state; feed tokens in blocks of 8, left-padding only the first partial block.

Accuracy

Validated against the reference PyTorch implementation (fp32) on the same prompt:

Export	logit correlation	max abs diff	top-5	greedy output
fp16	0.999999	0.025	5/5 identical	identical
q4f16	0.980	4.59	3/5 identical	identical

Usage (onnxruntime-web / onnxruntime-node)

const session = await ort.InferenceSession.create(modelBuffer, {
  executionProviders: ['webgpu'],
  externalData: [{ path: 'model.onnx.data', data: dataBuffer }],
});

const CHUNK = 8, L = 24, C = 2048, H = 32, N = 64;
let state = {
  state_att: new ort.Tensor('float32', new Float32Array(L * C), [L, C]),
  state_wkv: new ort.Tensor('float32', new Float32Array(L * H * N * N), [L, H, N, N]),
  state_ffn: new ort.Tensor('float32', new Float32Array(L * C), [L, C]),
};

// one block: left-pad to CHUNK, mask marks the real tokens
const ids = [...Array(CHUNK - block.length).fill(0), ...block];
const mask = [...Array(CHUNK - block.length).fill(0), ...block.map(() => 1)];
const out = await session.run({
  tokens: new ort.Tensor('int64', BigInt64Array.from(ids.map(BigInt)), [CHUNK]),
  mask: new ort.Tensor('float32', Float32Array.from(mask), [CHUNK]),
  ...state,
});
state = { state_att: out.state_att_out, state_wkv: out.state_wkv_out, state_ffn: out.state_ffn_out };

Prompt format:

System: <instructions>

User: <text>

Assistant:

Generation stops at token 0 (end of document) or at the next \n\n.

Conversion

Converted with the export script in SubForge AI, which builds the ONNX graph directly from the source checkpoint — weights are streamed to external data, and 4-bit packing uses onnxruntime's own MatMulNBits kernel.

License

Apache-2.0. This repository changes only the serialization format and numeric precision of the underlying weights.