Not-really-but-originally-Olmo-3 architecture 10M model with 2:1 SWA:GQA and HoPE embeddings. Trained on like 17B tokens of Fineweb-Edu (filtered to the splits before ChatGPT's release), DCLM-Edu and UltraData-Math.
(arch stability experiment more than anything)
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | ↑ | 0.1519 | ± | 0.0105 |
| none | 0 | acc_norm | ↑ | 0.2108 | ± | 0.0119 | ||
| arc_easy | 1 | none | 0 | acc | ↑ | 0.3822 | ± | 0.0100 |
| none | 0 | acc_norm | ↑ | 0.3636 | ± | 0.0099 | ||
| hellaswag | 1 | none | 0 | acc | ↑ | 0.2709 | ± | 0.0044 |
| none | 0 | acc_norm | ↑ | 0.2721 | ± | 0.0044 | ||
| piqa | 1 | none | 0 | acc | ↑ | 0.5582 | ± | 0.0116 |
| none | 0 | acc_norm | ↑ | 0.5555 | ± | 0.0116 |
Loading weights: 100% 132/132 [00:00<00:00, 9362.76it/s]
9,851,456 parameters (torch.bfloat16)
arithmark-3: 100% 32/32 [00:01<00:00, 20.44it/s]
arithmark-3: raw 33.00% (330/1000) normalized 33.10% (331/1000)
speed: 638.8 examples/s (tokenize 0.17s, evaluate 1.57s)
Category N Raw Normalized
----------------------------------------------------------------------------------------------
elementary_school_math_continuation::addition::grades_1_2::easy 128 28.91% 28.91%
elementary_school_math_continuation::comparison::grades_2_3::medium 44 20.45% 20.45%
elementary_school_math_continuation::comparison_difference::grades_2_3::medium 48 27.08% 27.08%
elementary_school_math_continuation::data::grades_2_3::easy 43 30.23% 32.56%
elementary_school_math_continuation::division::grades_3_4::medium 54 33.33% 31.48%
elementary_school_math_continuation::fractions_counting::grades_3_4::medium 50 16.00% 16.00%
elementary_school_math_continuation::geometry_area::grades_4_5::medium 52 53.85% 53.85%
elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium 45 37.78% 37.78%
elementary_school_math_continuation::measurement::grades_2_3::easy 76 34.21% 34.21%
elementary_school_math_continuation::money::grades_3_4::medium 64 28.12% 28.12%
elementary_school_math_continuation::multiplication::grades_3_4::medium 74 39.19% 39.19%
elementary_school_math_continuation::patterns::grades_3_4::medium 53 20.75% 20.75%
elementary_school_math_continuation::subtraction::grades_1_2::easy 117 24.79% 24.79%
elementary_school_math_continuation::time::grades_2_3::easy 55 98.18% 98.18%
elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium 46 21.74% 21.74%
elementary_school_math_continuation::two_step_addition::grades_2_3::medium 19 21.05% 21.05%
elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium 32 18.75% 21.88%
====================================================================
../exported_hf (9,851,456 params) RESULTS
====================================================================
Raw continuation accuracy 33.00%
Length-normalized accuracy 33.10%
Primary (acc_norm) 33.10%
====================================================================
- Downloads last month
- 14
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support