tomaarsen/colmodernvbert-st

ColModernVBERT

Model

This is the model card for ColModernVBERT, the late-interaction version of ModernVBERT that is fine-tuned for visual document retrieval tasks, our most performant model on this task.

Table of Contents

1. Overview
2. Usage
3. Evaluation
4. License
5. Citation

Overview

The ModernVBERT suite is a suite of compact 250M-parameter vision-language encoders, achieving state-of-the-art performance in this size class, matching the performance of models up to 10x larger.

For more information about ModernVBERT, please check the arXiv preprint.

Models

• ColModernVBERT is the late-interaction version that is fine-tuned for visual document retrieval tasks, our most performant model on this task.
• BiModernVBERT is the bi-encoder version that is fine-tuned for visual document retrieval tasks.
• ModernVBERT-embed is the bi-encoder version after modality alignment (using a MLM objective) and contrastive learning, without document specialization.
• ModernVBERT is the base model after modality alignment (using a MLM objective).

Usage

Sentence Transformers

ColModernVBERT can be loaded as a multi-vector (ColBERT-style late interaction) retriever with Sentence Transformers via the MultiVectorEncoder, exposing the familiar encode_query / encode_document / similarity API. This is the LoRA adapter repository, so the adapter is merged onto ModernVBERT/colmodernvbert-base at load time by a small modeling_st_colmodernvbert.py; trust_remote_code=True and peft are therefore required.

The MultiVectorEncoder class is recent and may not yet be in a stable Sentence Transformers release. Until it lands (in a version above 5.6.0), install from source:

pip install peft "sentence-transformers[image] @ git+https://github.com/UKPLab/sentence-transformers.git"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("ModernVBERT/colmodernvbert", trust_remote_code=True)

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
images = [
    "https://huggingface.co/ModernVBERT/colmodernvbert/resolve/main/assets/doc1.jpg",
    "https://huggingface.co/ModernVBERT/colmodernvbert/resolve/main/assets/doc2.jpg",
    "https://huggingface.co/ModernVBERT/colmodernvbert/resolve/main/assets/doc3.jpg",
    "https://huggingface.co/ModernVBERT/colmodernvbert/resolve/main/assets/doc4.jpg",
]

query_embeddings = model.encode_query(queries, convert_to_tensor=True)
document_embeddings = model.encode_document(images, convert_to_tensor=True)
print(f"Query 0 shape:    {tuple(query_embeddings[0].shape)}")
print(f"Document 0 shape: {tuple(document_embeddings[0].shape)}")
# Query 0 shape:    (26, 128)
# Document 0 shape: (1149, 128)

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[16.7640, 10.4327, 11.8267,  9.0278],
#         [ 7.3852, 12.0147,  8.1190,  7.9530]])

The adapter loads in float32 by default; pass model_kwargs={"torch_dtype": "bfloat16"} for lower memory. The query augmentation (10 <end_of_utterance> tokens) and the image visual prompt (<|begin_of_text|>User:<image>Describe the image.<end_of_utterance>\nAssistant:) are baked into chat_template.jinja, so encode_query and encode_document reproduce ColModernVBertProcessor.process_queries / process_images exactly. To reproduce colpali-engine's mask_non_image_embeddings=True (scoring only the image-patch tokens), enable the mask allowlist after loading:

model[-1].keep_only_token_ids = [50407]  # the <image> token id

sentence_transformers.multi_vector_encoder.interpretability.get_n_patches raises NotImplementedError for this model: like other Idefics3-style split-image processors, each page is split into sub-patch token blocks plus a global patch, so the token grid is not a simple rectangle.

ColPali Engine

🏎️ If your GPU supports it, we recommend using ModernVBERT with Flash Attention 2 to achieve the highest GPU throughput. To do so, install Flash Attention 2 as follows, then use the model as normal:

For now, the branch for using colmdernvbert is not yet merged in the official colpali repo, you need to clone the repo and checkout on the right branch to use it.

pip install colpali-engine

Here is an example of masked token prediction using ModernVBERT:

import torch
from colpali_engine.models import ColModernVBert, ColModernVBertProcessor
from PIL import Image
from huggingface_hub import hf_hub_download

model_id = "ModernVBERT/colmodernvbert"

processor = ColModernVBertProcessor.from_pretrained(model_id)
model = ColModernVBert.from_pretrained(
            model_id,
            torch_dtype=torch.float32, # use torch_dtype=torch.bfloat16 for flash attention
            trust_remote_code=True
)

image = Image.open(hf_hub_download("HuggingFaceTB/SmolVLM", "example_images/rococo.jpg", repo_type="space"))
text = "This is a text"

# Prepare inputs
text_inputs = processor.process_texts([text])
image_inputs = processor.process_images([image])

# Inference
q_embeddings = model(**text_inputs)
corpus_embeddings = model(**image_inputs)

# Get the similarity scores
scores = processor.score(q_embeddings, corpus_embeddings)

print("Similarity scores:", scores)

Evaluation

ColModernVBERT matches the performance of models nearly 10x larger on visual document benchmarks. Additionally, it provides an interesting inference speed on CPU compared to the models of similar performance.

License

We release the ModernVBERT model architectures, model weights, and training codebase under the MIT license.

Citation

If you use ModernVBERT in your work, please cite:

@misc{teiletche2025modernvbertsmallervisualdocument,
      title={ModernVBERT: Towards Smaller Visual Document Retrievers}, 
      author={Paul Teiletche and Quentin Macé and Max Conti and Antonio Loison and Gautier Viaud and Pierre Colombo and Manuel Faysse},
      year={2025},
      eprint={2510.01149},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2510.01149}, 
}