Update cagliostro-v3 to final weights (75B tokens)
Thanks for merging #157 and for re-running the numbers.
That entry was a mid-training checkpoint at 72.745B tokens. The run has since finished at 75.00B tokens, 762,939 steps, learning rate fully decayed to zero, so this updates the row to the final weights.
| Benchmark | merged (72.7B) | final (75B) |
|---|---|---|
| ARC-Easy | 54.17 | 54.88 |
| ARC-Challenge | 27.99 | 28.75 |
| HellaSwag | 42.42 | 42.51 |
| PIQA | 67.46 | 67.46 |
| ArithMark-3 | 43.80 (mine) / 43.20 (yours) | 43.70 (mine) |
On ArithMark I measured 43.80 on the old checkpoint against your 43.20. I have not been able to work out where the 0.6 comes from and I am not disputing it, your number is the one that counts. I have put my 43.70 in the row for the final weights, so please overwrite it with whatever you measure.
Model, with final weights, updated card and training charts: https://huggingface.co/bench-labs/cagliostro-v3
python -m lm_eval --model hf \
--model_args pretrained=bench-labs/cagliostro-v3,dtype=float32,trust_remote_code=True \
--tasks hellaswag,arc_easy,arc_challenge,piqa \
--num_fewshot 0 --batch_size 8 --device cuda:0
float32 matters here, the model has a logit cap of 15 and a bfloat16 round trip moves logits about 1.3e-1, enough to flip borderline answers.
0.6 probably comes from natural variation, like seeds and such idk
good benches