Spaces:

---

multimodalart

/

jev-decision-index

Like 851

Running

Decision Index 0.4 lists d3-turbo from the first version of PR #242 (v1.0 = da17b5d0, public 71.61). This request and the PR now pin the final version, v1.1 = 689b3af1 (public 71.62); please score v1.1 in the next batch, for the text and vision boards, in place of the v1.0 entry.

Hi @multimodalart ! vLLM-SR d3-turbo (Decision 3.0, 27B) is submitted to the text board with its complete public 0.3 run (the pull request is linked in the comment below). It also reads images, and we would like to ask for a Vision board evaluation. It is pinned to the final version of the model, v1.1.

It is public (Apache-2.0), and this is its code path:

• Code (in the repo, same revision): the repo's own runtime: modeling_d3.py (AutoModel.from_pretrained("vllm-sr/d3-turbo", trust_remote_code=True)), d3_runtime.py, the kit engine d3_engine:D3Engine (engine(state, questions, images=[...])) and a /v1/systemone server d3_server.py (request body images: [...]).
• Readout: the input is read once per request (state once), and every question is answered from it: each option sits under a single-token answer code, read out at the last prompt position, giving one probability per option (no text is generated).
• Image contract: multiple images per request, within the 32,768-token limit (local paths, http(s) URLs, PIL images or base64 data URLs; the server takes base64 PNG/JPEG/WebP data URLs of at most 8 MB and 16 MP), placed before the text of the user turn; the model's own processor resizes each image to at most 1.6 MP; every question of the request sees all of its images. Images can also sit inside a question or an option, where only that question sees them. Up to 255 options per question; 32,768 tokens per question by default.
• No Decision Index evaluation items (text or vision) were used in training.

Thank you!