Sazanami-Embed-310m

English | 日本語

A Japanese single-vector retrieval model (bi-encoder) trained from sbintuitions/modernbert-ja-310m. It encodes queries and documents each into a single 768-dimensional vector, and compares them with cosine similarity. It can process up to 8,192 tokens.

This model is paired with the late-interaction model Sazanami-ColBERT-310m, built with the same initialization, training data, and loss. Because the only difference is how many vectors represent a document, multi-vector and single-vector representations can be compared without changing any other condition.

1. Model Overview

Property Value
Base model sbintuitions/modernbert-ja-310m
Architecture Transformer → mean pooling (including prefix tokens)
Parameters About 315M
Vector dimensions 768
Number of vectors per document 1
Maximum input length 8,192 tokens (default stored in the model; training was conducted with 512 tokens)
Scoring function Cosine similarity
Prefix 検索クエリ: ("search query: ") for queries, 検索文書: ("search document: ") for documents
Tokenizer SentencePiece (uses the ModernBERT-Ja vocabulary as-is)
Main framework sentence-transformers (SentenceTransformer)
Mixed precision during training bf16

Related Models

All four models were trained from sbintuitions/modernbert-ja-310m on the same training data. What differs is how many vectors represent a single document. This model is a single-vector model, trained under the same conditions for comparison with the three multi-vector models.

Model Vectors per document When to choose
Sazanami-ColBERT-310m Same as the number of non-padding tokens in the text Highest accuracy when searching for long documents among many candidates
Sazanami-AGC-310m Up to 32 When you want to reduce storage size
Sazanami-MetaEmbed-310m Up to 64 (the number used can be reduced) When you want faster similarity computation
Sazanami-Embed-310m (this model) 1 (768 dimensions) When you want to minimize storage size and computational cost

2. Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("mahiyama/Sazanami-Embed-310m")

# Queries: "Where can I get a certificate of residence?", "I want to know the enrollment procedure for National Health Insurance"
queries = [
    "住民票はどこで取れますか",
    "国民健康保険の加入手続きを知りたい",
]
# Documents: on obtaining a certificate of residence, National Health Insurance enrollment, and oversized garbage collection
documents = [
    "住民票の写しは、市区町村の窓口またはコンビニ交付サービスで取得できます。",
    "国民健康保険の加入手続きは、転入や退職から 14 日以内に市区町村の窓口で行ってください。",
    "粗大ごみの収集は、事前の申し込みが必要です。",
]

# Prefixes are stored in the model, so do not add them to the text yourself
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)

# Scored with cosine similarity according to the stored settings
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)

3. Training

Base Model

The model is initialized with sbintuitions/modernbert-ja-310m. It has undergone only masked language modeling pre-training and has not been trained for retrieval. The architecture simply adds mean pooling to this Transformer, with no additional projection layer.

Training Data

Two formats were used: n-tuples, pairing one query with one positive and five negatives, and pairs, pairing queries with only a positive.

Source Format Count
mahiyama/auto-wiki-qa n-tuples 100,000
mahiyama/mqa-ja n-tuples 100,000
mahiyama/mmarco-ja n-tuples 50,000
mahiyama/civicqa-ja (non-public) n-tuples 43,382
mahiyama/quiz-no-mori n-tuples 13,422
mahiyama/quiz-works n-tuples 12,502
mahiyama/amagasaki-qna n-tuples 11,069
mahiyama/miracl-retrieval n-tuples 4,603
mahiyama/mrtydi n-tuples 3,602
mahiyama/anlp-meeting-retrieval (title-abs) n-tuples 1,926
mahiyama/anlp-meeting-retrieval (title-intro) n-tuples 1,952
mahiyama/civicqa-ja (non-public) pairs 28,653
mahiyama/amagasaki-qna pairs 7,323
mahiyama/kosodate-faq-pairs-ja pairs 595
Total 379,029

mahiyama/civicqa-ja consists of questions and answers collected from FAQs published by Japanese government agencies and local governments. It is currently non-public.

Data used for evaluation was excluded from training. JaGovFaqs-22k was not used in training at all. Rows where the query after NFKC normalization and whitespace removal matched a query in the evaluation split of JaGovFaqs-22k, or where the positive after the same processing matched a document in that evaluation split, were excluded at each stage.

Training Procedure

For n-tuples, the sum of a contrastive loss and a distillation loss is used. Both are computed from a single score computation.

  • Contrastive loss — Cross-entropy with all documents in the batch (its own 6 candidates and documents from other rows in the same batch) as candidates. Cosine similarity is multiplied by 20 before use.
  • Distillation loss — KL divergence between the probability distribution of cosine similarities over its own 6 candidates and the probability distribution of teacher scores. The teacher temperature is 2.0.

For pairs, a contrastive loss is used with documents from other rows in the batch as negatives.

Setting Value
learning_rate 2e-5
num_train_epochs 1
per_device_train_batch_size 64
scale (multiplier for cosine similarity) 20.0
warmup_steps 5% of total
weight_decay 0.01
Maximum input length during training 512 tokens
Stored maximum input length for inference 8,192 tokens
Mixed precision bf16
batch_sampler NO_DUPLICATES
Gradient checkpointing Enabled
Teacher temperature 2.0
Random seed 42

Training took about 4 hours 27 minutes (5,923 steps) on a single NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB) GPU, with peak GPU memory usage of 22.1 GB.

4. License and Training Data Restrictions

The model weights are released under the MIT License. The base model, sbintuitions/modernbert-ja-310m, is also released under the MIT License.

The training data includes a Japanese translation of MS MARCO (mahiyama/mmarco-ja). MS MARCO is released for non-commercial research purposes, and its license for commercial use is not clear. If you use this model commercially, please check the MS MARCO terms of use at your own responsibility.

Part of the training data includes questions and answers collected from FAQs published by Japanese government agencies and local governments. This data is subject to the terms of use of each source and is not redistributed in this model's repository. If you use this model, please determine the suitability for your application at your own responsibility.

Downloads last month
58
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for mahiyama/Sazanami-Embed-310m

Finetuned
(25)
this model

Datasets used to train mahiyama/Sazanami-Embed-310m

Collection including mahiyama/Sazanami-Embed-310m