Model card and blog: VL-RewardBench #1 (78.3%)

README.md

@@ -37,6 +37,25 @@ serves both systems, for text and for images:
 JEV-27B's language weights are bit-identical to Qwen3.8-27B's, so on text this model gives the same System 1 decisions as
 JEV-27B. On 24 reference decisions the largest probability difference was 0.024 (bf16 noise), with the same top option 24/24.
 
+## Highlight: #1 on the VL-RewardBench leaderboard (multimodal judge)
+
+Given an image, a question and two answers, System 1 says which answer is better, zero-shot and in one forward pass per order.
+On [VL-RewardBench](https://vl-rewardbench.github.io) (CVPR 2025, 1,247 human-verified pairs) it reaches **78.3% overall
+accuracy**, the highest on the official leaderboard (26 models, updated May 2025):
+
+| judge | general | hallucination | reasoning | **overall** | macro |
+|---|---:|---:|---:|---:|---:|
+| **JEV-27B-VL System 1** | 58.0 | **83.2** | **78.2** | **78.3** | **73.1** |
+| Skywork-VL-Reward-7B | **65.6** | 80.2 | 61.3 | 73.3 | 69.0 |
+| Gemini 2.0 Flash | 50.8 | 72.6 | 70.1 | 68.8 | 64.5 |
+| Gemini 1.5 Pro | 50.8 | 72.5 | 64.2 | 67.2 | 62.5 |
+| GPT-4o | 49.1 | 67.6 | 70.5 | 65.8 | 62.4 |
+| Claude 3.5 Sonnet | 43.4 | 55.0 | 62.3 | 55.3 | 53.6 |
+
+2,494 decisions in 163 seconds on one GPU. Code: [JEV-27B-DEMO / 09-multimodal-judge](https://github.com/yuhai-china/JEV-27B-DEMO/tree/master/09-multimodal-judge).
+
+![multimodal judge](multimodal_judge.png)
+
 ## Highlight: zero-shot image recommendation
 
 System 1's decision head was trained on text only. Given images, it still works zero-shot. On
@@ -107,6 +126,7 @@ Human gold labels (CLINC-150, MTOP, GoEmotions, DBpedia), 800 items.
 | task | result | comparison |
 |---|---|---|
 | **Image recommendation, zero-shot** (MicroLens, covers only) · JEV-27B-VL | **AUC 0.727** | collaborative filtering from 59,045 users' logs 0.728 · titles only 0.649 |
+| **Multimodal judge** (VL-RewardBench, 1,247 pairs) · JEV-27B-VL | **78.3% accuracy**, #1 on the leaderboard | Skywork-VL-Reward-7B 73.3 · GPT-4o 65.8 · Claude 3.5 Sonnet 55.3 |
 | **Biomedical research questions** (PubMedQA test, 500) · JEV-27B-VL | **77.8% accuracy**, zero-shot | human experts 78.0% · GPT-4 zero-shot 75.2% · BioBERT 68.1% |
 | **Response judge** (RewardBench, 2,985 pairs) | **89.9** | Gemini 1.5 Pro 88.2 · GPT-4o 86.7 · Claude 3.5 Sonnet 84.2 as judges |
 | **Hallucination guard** (TriviaQA, answer the trusted half) | **96.4% accuracy** | answering everything 71.2% · model's own stated confidence 87.2% |

blog/JEV-27B-VL.md

@@ -11,6 +11,9 @@ recommender built from tens of thousands of users' behaviour.
 
 **TL;DR**
 
+* **The best multimodal judge on VL-RewardBench:** shown an image, a question and two answers, it picks the better
+  answer **78.3%** of the time, the highest on the official leaderboard, ahead of the purpose-trained Skywork-VL-Reward-7B
+  (73.3%), Gemini 2.0 Flash (68.8%) and GPT-4o (65.8%). Best of all at catching visual hallucinations (83.2%).
 * **Zero-shot image recommendation:** looking only at video covers, JEV-27B-VL ranks what a user will watch next as well as
   collaborative filtering learned from **59,045 users' watch histories** (AUC 0.727 vs 0.728), with a higher top-5 hit rate
   (59% vs 49%), and without any interaction data.
@@ -30,6 +33,29 @@ recommender built from tens of thousands of users' behaviour.
 Nothing to parse and no prompt gymnastics: you ask *"Is this scenario one where the user clicks this video?"* and get
 `{"true": 0.83, "false": 0.17}`.
 
+## It judges images better than anything on the leaderboard
+
+Training and evaluating vision-language models needs a judge: of two answers about an image, which one is right, and which one
+describes things that are not there? [VL-RewardBench](https://vl-rewardbench.github.io) (CVPR 2025) tests exactly that with
+1,247 human-verified pairs: real users' questions, visual hallucination detection, and multimodal knowledge and math. Its
+authors note that even GPT-4o gets only about two thirds right.
+
+JEV-27B-VL answers each pair in one forward pass per ordering, with no training on images:
+
+![VL-RewardBench](https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/multimodal_judge.png)
+
+| judge | general | hallucination | reasoning | **overall** |
+|---|---:|---:|---:|---:|
+| **JEV-27B-VL** | 58.0 | **83.2** | **78.2** | **78.3** |
+| Skywork-VL-Reward-7B (trained reward model) | **65.6** | 80.2 | 61.3 | 73.3 |
+| Gemini 2.0 Flash | 50.8 | 72.6 | 70.1 | 68.8 |
+| Gemini 1.5 Pro | 50.8 | 72.5 | 64.2 | 67.2 |
+| GPT-4o | 49.1 | 67.6 | 70.5 | 65.8 |
+| Claude 3.5 Sonnet | 43.4 | 55.0 | 62.3 | 55.3 |
+
+That is the highest overall accuracy on the official leaderboard (26 models, last updated May 2025): 5 points above the best
+purpose-trained reward model and 12.5 points above GPT-4o. All 2,494 judgements took 163 seconds on one GPU.
+
 ## It sees: zero-shot image recommendation
 
 We tested it on [MicroLens](https://github.com/westlake-repl/MicroLens), a public dataset of real users of a short-video app