Telugu Tokenizer Leaderboard
🔤
How many tokens does a Telugu word cost? Compare tokenizers
అ Akshara
English + Telugu LLMs built end to end: data engine, tokenizer, pretraining, serving. Start with the Telugu tokenizer leaderboard.
Note 27 tokenizers on Telugu and English. Paste your own text; see why the pre-tokenization rule decides most of the gap.
kuluruvineeth/akshara-tokenizer-v0
UpdatedNote 65,536-token English + Telugu BPE: 1.69 tokens per Telugu word on FLORES-200, 1.32 for English.
kuluruvineeth/common-crawl-20-warcs-en-te
Viewer • Updated • 378k • 43Note What the data engine keeps and removes on 20 Common Crawl WARCs, every removed page browsable by reason. $0.04 on HF Jobs.
dataflow runs
📈Live dashboards of dataflow pipeline runs on HF Jobs
Note Live Trackio dashboards of data-engine runs: throughput, cost so far, the removal funnel.