regnant-io/kw5-109M

KW5 109M

A 109.5M-parameter Swahili (Kiswahili) base language model, pretrained from scratch on 1.41B tokens on a single NVIDIA T4. This is the first KW5 model. Its successor is kw5-149M.

Built by Regnant.

This is a base model. It does next-token prediction only. The instruction tuned version is kw5-109M-instruct.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("regnant-io/kw5-109M")
tok = AutoTokenizer.from_pretrained("regnant-io/kw5-109M")

# A base model continues a prefix; it does not follow instructions.
inputs = tok("Tanzania ni nchi", return_tensors="pt", add_special_tokens=False)
out = model.generate(**inputs, max_new_tokens=60)
print(tok.decode(out[0], skip_special_tokens=True))

Output with torch.manual_seed(0) on CPU (sampling is on, so yours will differ):

Tanzania ni nchi ya Afrika Mashariki na Kati. Kwa hiyo, kwa kuwa tunapakana na
Kenya, Uganda, Rwanda, Burundi, DRC Kongo (DRC), Zambia, Malawi, Msumbiji,
Mauritius, Seychelles, Mozambique, Swaziland, Zimbabwe, Comoro, ...

Fluent, and wrong: Mauritius, Seychelles, Zimbabwe and Swaziland do not border Tanzania. That is typical of this model.

No <s> is needed. Pretraining packed documents without a leading BOS. An earlier revision of this card said every training sequence began with <s>; it did not. Adding one anyway is harmless: on held-out text it moves the loss by 0.009 nats.

The shipped generation_config.json samples at temperature 0.2, top-p 0.9, repetition penalty 1.3, which is what the output above used.

Model details

Parameters	109.5M (tied input and output embeddings)
Architecture	Llama-compatible decoder-only transformer
Layers / hidden / FFN	12 / 768 / 2048
Attention	12 query heads, 12 key-value heads (standard multi-head attention, not GQA)
Normalization / activation	RMSNorm, pre-norm / SwiGLU
Position encoding	RoPE, theta 10,000
Trained context	1024 tokens
max_position_embeddings	2048, but see below
Vocabulary	32,000 SentencePiece BPE, NFC, digits split, byte fallback
Precision	FP32 weights

Context: use 1024 tokens

The config allows 2048 positions, but the model was only ever trained at 1024. The planned extension to 2048 was gated behind a decay phase that never ran (see Training), so rope_theta is still 10,000. Measured loss by position on 12 long Swahili documents:

positions	0 to 511	512 to 1023	1024 to 1535	1536 to 2047
loss (nats)	4.12	4.08	4.45	5.92

Past 1024 tokens the model is reading positions it never saw. Keep prompt plus output under 1024.

Tokenizer

32k BPE trained from scratch on the deduplicated Swahili corpus. Special ids: <unk>=0, <s>=1, </s>=2, <pad>=3. Ids 4 to 6 (<|system|>, <|user|>, <|assistant|>) are reserved but never occur in the pretraining corpus, so their embedding rows are untrained. If you fine-tune with them as role markers, train embed_tokens too.

Training

Data	FineWeb-2 swh_Latn, cleaned and deduplicated (exact and MinHash LSH), about 705M unique tokens
Tokens seen	1.41B, two passes over the corpus
Checkpoint	step 6,150
Packing	documents concatenated with no attention mask at document boundaries
Precision	FP16 mixed precision with GradScaler
Optimizer	8-bit AdamW, lr 3e-4, weight decay 0.1, grad clip 1.0
Schedule	Warmup-Stable-Decay, stopped in the stable phase
Hardware	1 NVIDIA T4, about 60 GPU-hours in resumable sessions

The learning rate was never decayed. The token budget was set to 8B from a corpus-size estimate that turned out to be about four times too high, so the decay phase was scheduled for a point the run never reached. Training stopped at 1.41B tokens with the learning rate still at 3e-4. There was no held-out validation set in this run. An earlier revision of this card said the decay was skipped on purpose because validation quality was falling; that was not what happened.

Because packing had no document mask, the model was trained to attend across unrelated documents inside one sequence.

Evaluation

Loglikelihood multiple choice, n = 50 per task:

task	accuracy	95% interval	chance
Belebele-sw (4-way)	32%	21 to 46	25%
AfriXNLI-sw (3-way)	32%	21 to 46	33%

Both are consistent with chance. With 50 items this is a floor, not a capability. The model's one real skill is fluent Swahili continuation.

Intended use

A small starting point for Swahili experiments and fine-tuning. For new work, start from kw5-149M instead.

Not for deployment: no instruction following, no safety tuning, poor factual reliability.

Limitations

• It continues text; it does not answer questions.
• Facts are unreliable, as the sample above shows.
• Mostly Tanzanian Swahili web text, with the biases of web text.
• No safety tuning at all.

Files

file	what it is
model.safetensors	FP32 weights, tied embeddings
config.json	Llama config, pad_token_id 3
tokenizer.model, tokenizer.json	SentencePiece 32k BPE, and the same as a fast tokenizer
tokenizer_config.json, special_tokens_map.json	add_bos_token: false
generation_config.json	temperature 0.2, top-p 0.9, repetition penalty 1.3
training_info.json	checkpoint step, tokens seen, the n = 50 scores above

Citation

@misc{kw5_109m_2026,
  title  = {KW5 109M: a Swahili base language model trained on a single T4},
  author = {Regnant},
  year   = {2026},
  url    = {https://huggingface.co/regnant-io/kw5-109M}
}

Apache 2.0.