Jitesh K Prasad PRO

@chronossystemic Thanks, and I agree. Bits per byte going down does not mean the model knows more. On the knowledge and reasoning tests Vemana is at chance, like most open models under 1B we tested, and we say that on the card.

All our held-out numbers are on fresh Telugu Wikipedia pages, so outside Wikipedia we have not measured it. All 12 checkpoints are public, so anyone can track bits per byte and a fixed task side by side across the run, on whatever Telugu text they care about:
https://huggingface.co/udgamlabs/Vemana-Checkpoints

@dipankarsarkar Thanks for running it, and your 65.9% and 56.2% match ours exactly. I did not know the main → में miss comes from the word list and not from the missing context. So our card is not fully right there, the model alone gets मैं, and it is the list that pulls it to में.

And the top-2 point is a good one. tr.word() already gives all 8 candidates, so a keyboard can show the runner-up as a chip, and by your count that takes it to 79.0%.

On your question, no, we have not tried a two-word prior. Context from the previous word was on our list of possible fixes, but we never tested it, so I do not know yet what it would cost in size. Everything is open, the model, the 200,000-word list and the runner, so anyone who wants to try it on top can.

@Kreout You are right, and thanks for this. I just checked, right now tr.text() changes every run of Latin letters, so name@example.com comes out as नाम@एग्जाम्पल.कॉम, and a URL or a user_id gets changed the same way. In a real text field that would surprise people.

For now, if you are building with it, you can keep URLs, emails and IDs as they are on your side and call tr.word() only on the words. The runner is one small JS file and it is all open, so it is easy to add that rule. And I agree this belongs on the card, next to the accuracy.

@Twu31 Good point, and sorry for the late reply. No, we have not tried that format. Right now the two formats change two things at once, how the answer is scored and whether the model sees the other choices, so the cloze minus letter gap mixes both. I think your version would separate them.

One thing we found after the post: 19 of the 27 models pick one favourite letter on more than half the questions. Gargi picks B on 94% of them. So a big part of the letter score is just which letter a model likes, but without your third format we cannot say how much of the gap is the hidden choices.

Every model version is pinned on the board, and the dataset has the per-question results for 25 of the 27, so anyone can run it and compare question by question:
https://huggingface.co/datasets/udgamlabs/small-model-board-scores

@8BitStudio Sorry, I missed this one. Yes, I think you are right, at this size the data is basically everything. For Gargi it was openly licensed and public-domain text plus synthetic text from open models, the details are on the card. But I agree, at 17B tokens what you pick matters a lot more than how much you have.

And also your result is really interesting, division and carrying at 18B tokens. If you put that model out, I would love to try it.

@dipankarsarkar Yes, our numbers match yours, SmolLM2-360M +0.97 and Gargi +0.17. And you are right, the four letter shares are enough to check the own-mix gap, so anyone can check nltower without its per-question rows. I like that a lot, because then every number on the board is checkable.

I have to be honest on one thing. The board does not have these columns yet. We built the favourite letter, the own-mix baseline and the gap for all 27 models on our side, but that update is not live.

And thanks again for all the work on this. I think you found more in our scores than we did.

@dipankarsarkar You are right, the own-mix null is the correct one. We checked it on our side. SmolLM2-360M goes from +2.6 against its favourite letter to +0.9 against its own mix, and Gargi stays at about +0.2 either way. Against own mix only LFM2.5 (+13.3), the two Falcons (+8.2 and +7.9) and nltower (+3.9, we have its rows but cannot publish them) clear 2 points. So I think the own-mix gap is the column to show, not the share.

Rotation is not planned right now. But when we do it, yes, we will keep the unrotated column next to it, because like you said the gap between them is the bias.

@dipankarsarkar Thanks for actually running the parquet. We re-counted from the raw logs with the real answer key, and your numbers hold exactly. Gargi picks B on 94.1% of the questions and scores 24.89, against 24.65 for always answering B. Gargi-Base is 66.1% B, and accuracy on the non-B questions goes from 11.8 to 2.0 after SFT. So you are right, SFT did move something, it just pushed the model onto B.

I think the cleanest way to see it is each model against its own favourite letter. Only 5 of the 27 beat always answering their favourite letter by more than 2 points: LFM2.5-230M (+13.3), the two Falcon-H1-Tiny models (+9.5 and +8.5), nltower (+3.7) and SmolLM2-360M (+2.6). The other 22, including all of ours, are less than 2 points away from it. So for most of these models the letter score is basically just which letter they like.

On your question, I think rotating the options is the cleaner fix, because then a model stuck on one letter gets exactly 25 whichever letter it is. Off-modal accuracy puts each model on a different set of questions. For now we will add the favourite-letter share to the board, so anyone can see which letter each model leans on.

Would you also show each model's score minus its own favourite letter, or is the share enough?

@RakshithFury Yes, I think you are right about the thinking mode. At this size the gain is small. Gargi gets 32.9 on GSM8K with thinking off and 36.0 with it on, so about 3 points. On our smaller model Laghu (152M) it helped only on short arithmetic, and it actually made instruction following and multi-turn chat worse. We kept it mainly because I wanted the full package of a bigger model (thinking switch, multi-turn, tool format, 16K context) in something that runs on a laptop. But that is not where the quality comes from.

And I agree on RL also. Our Laghu try is one result with a weak base model, so it does not show that RL cannot work below 500M. If the base is good enough, it should help.

On your question: yes, basically. We are a small lab and we fund it ourselves, so a low budget is not really a choice for us. But I think it is also a useful question by itself, how far a small model can go if you are careful with the data and the evals. That is why we run every comparison ourselves and put the per-question results out.