Self-scored Decision Index 0.3 run for Xor 26B-A4B v1.2 (juspay/Xor-26B-A4B): 59.46
Following on from #36, we've released Xor 26B-A4B v1.2 (https://huggingface.co/juspay/Xor-26B-A4B/tree/v1.2, tag v1.2, BF16, Apache-2.0). We scored it on the public Decision Index 0.3 suite with the kit and opened a pull request on apolinario/decision-index: https://github.com/apolinario/decision-index/pull/202
- Result: 59.46 public index (raw 69.41, breadth 58.76). The run is complete (140,178 of 140,178 scoreable requests), with all 140,620 requests ok.
- Artifacts: the full run, scores and harness are at https://huggingface.co/datasets/juspay/xor-26b-a4b-v1.2-decision-index
- To run it: the model repo's serving/xor-v1.2-serving.tar.gz bundle (/v1/systemone, one GPU, quick start on the card). It served 131,072 tokens of context in our run.
@Prasad229 @sahebjot