Benchmark Summary
Gov Chatbot Red-Team v2
| Suite | Full F1 | Full Exact F1 | Full Failures | Q8 F1 | Q8 Exact F1 | Q8 Failures |
|---|---|---|---|---|---|---|
| Gov chatbot red-team v2 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 1.0000 | 0.0000 |
| Gov chatbot gap holdout v2 | 1.0000 | 1.0000 | 0.0000 | 1.0000 | 1.0000 | 0.0000 |
ONNX q8
| Suite | F1 | Examples/s |
|---|---|---|
| Irish core | 1.0000 | 221.5743 |
| UAT exact | 1.0000 | 188.6537 |
| Fresh holdout | 0.9841 | 145.9336 |
| Hardening exact | 1.0000 | 112.4948 |
| Edge | 1.0000 | 235.2888 |
| Multilingual PPSN overall | 0.9333 | 224.6955 |
| Multilingual PPSN label-only | 1.0000 | — |
Full checkpoint
| Suite | F1 | Examples/s |
|---|---|---|
| Irish core | 1.0000 | 79.8593 |
| UAT exact | 1.0000 | 29.1305 |
| Fresh holdout | 0.9841 | 31.4254 |
| Hardening exact | 1.0000 | 26.6732 |
| Edge | 1.0000 | 52.4220 |
| Multilingual PPSN overall | 0.9333 | 89.3536 |
| Multilingual PPSN label-only | 1.0000 | — |