8000 vocab size tokenizer padded to 8192, trained on the following data:

DATASETS = [
    ("mlfoundations/dclm-baseline-1.0-parquet", None, "train", "text", 75000),
    ("HuggingFaceFW/finepdfs-edu", "eng_Latn", "train", "text", 7500),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "cpp", "content", 5000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "js", "content", 20000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "py", "content", 20000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "rust", "content", 5000),
    ("openbmb/UltraData-Math", "UltraData-Math-L2-preview", "train", "content", 15000),
]

Code is in notebook.py

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

allura-forge
/

Rambley-Tokenizer-v1-8k

Datasets used to train allura-forge/Rambley-Tokenizer-v1-8k