Instructions to use mahiyama/Sazanami-Embed-310m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use mahiyama/Sazanami-Embed-310m with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("mahiyama/Sazanami-Embed-310m") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Notebooks
- Google Colab
- Kaggle
Sazanami-Embed-310m
English | 日本語
A Japanese single-vector retrieval model (bi-encoder) trained from sbintuitions/modernbert-ja-310m. It encodes queries and documents each into a single 768-dimensional vector, and compares them with cosine similarity. It can process up to 8,192 tokens.
This model is paired with the late-interaction model Sazanami-ColBERT-310m, built with the same initialization, training data, and loss. Because the only difference is how many vectors represent a document, multi-vector and single-vector representations can be compared without changing any other condition.
1. Model Overview
| Property | Value |
|---|---|
| Base model | sbintuitions/modernbert-ja-310m |
| Architecture | Transformer → mean pooling (including prefix tokens) |
| Parameters | About 315M |
| Vector dimensions | 768 |
| Number of vectors per document | 1 |
| Maximum input length | 8,192 tokens (default stored in the model; training was conducted with 512 tokens) |
| Scoring function | Cosine similarity |
| Prefix | 検索クエリ: ("search query: ") for queries, 検索文書: ("search document: ") for documents |
| Tokenizer | SentencePiece (uses the ModernBERT-Ja vocabulary as-is) |
| Main framework | sentence-transformers (SentenceTransformer) |
| Mixed precision during training | bf16 |
Related Models
All four models were trained from sbintuitions/modernbert-ja-310m on the same training data. What differs is how many vectors represent a single document. This model is a single-vector model, trained under the same conditions for comparison with the three multi-vector models.
| Model | Vectors per document | When to choose |
|---|---|---|
| Sazanami-ColBERT-310m | Same as the number of non-padding tokens in the text | Highest accuracy when searching for long documents among many candidates |
| Sazanami-AGC-310m | Up to 32 | When you want to reduce storage size |
| Sazanami-MetaEmbed-310m | Up to 64 (the number used can be reduced) | When you want faster similarity computation |
| Sazanami-Embed-310m (this model) | 1 (768 dimensions) | When you want to minimize storage size and computational cost |
2. Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("mahiyama/Sazanami-Embed-310m")
# Queries: "Where can I get a certificate of residence?", "I want to know the enrollment procedure for National Health Insurance"
queries = [
"住民票はどこで取れますか",
"国民健康保険の加入手続きを知りたい",
]
# Documents: on obtaining a certificate of residence, National Health Insurance enrollment, and oversized garbage collection
documents = [
"住民票の写しは、市区町村の窓口またはコンビニ交付サービスで取得できます。",
"国民健康保険の加入手続きは、転入や退職から 14 日以内に市区町村の窓口で行ってください。",
"粗大ごみの収集は、事前の申し込みが必要です。",
]
# Prefixes are stored in the model, so do not add them to the text yourself
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
# Scored with cosine similarity according to the stored settings
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
3. Training
Base Model
The model is initialized with sbintuitions/modernbert-ja-310m. It has undergone only masked language modeling pre-training and has not been trained for retrieval. The architecture simply adds mean pooling to this Transformer, with no additional projection layer.
Training Data
Two formats were used: n-tuples, pairing one query with one positive and five negatives, and pairs, pairing queries with only a positive.
| Source | Format | Count |
|---|---|---|
| mahiyama/auto-wiki-qa | n-tuples | 100,000 |
| mahiyama/mqa-ja | n-tuples | 100,000 |
| mahiyama/mmarco-ja | n-tuples | 50,000 |
| mahiyama/civicqa-ja (non-public) | n-tuples | 43,382 |
| mahiyama/quiz-no-mori | n-tuples | 13,422 |
| mahiyama/quiz-works | n-tuples | 12,502 |
| mahiyama/amagasaki-qna | n-tuples | 11,069 |
| mahiyama/miracl-retrieval | n-tuples | 4,603 |
| mahiyama/mrtydi | n-tuples | 3,602 |
| mahiyama/anlp-meeting-retrieval (title-abs) | n-tuples | 1,926 |
| mahiyama/anlp-meeting-retrieval (title-intro) | n-tuples | 1,952 |
| mahiyama/civicqa-ja (non-public) | pairs | 28,653 |
| mahiyama/amagasaki-qna | pairs | 7,323 |
| mahiyama/kosodate-faq-pairs-ja | pairs | 595 |
| Total | 379,029 |
mahiyama/civicqa-ja consists of questions and answers collected from FAQs published by Japanese government agencies and local governments. It is currently non-public.
Data used for evaluation was excluded from training. JaGovFaqs-22k was not used in training at all. Rows where the query after NFKC normalization and whitespace removal matched a query in the evaluation split of JaGovFaqs-22k, or where the positive after the same processing matched a document in that evaluation split, were excluded at each stage.
Training Procedure
For n-tuples, the sum of a contrastive loss and a distillation loss is used. Both are computed from a single score computation.
- Contrastive loss — Cross-entropy with all documents in the batch (its own 6 candidates and documents from other rows in the same batch) as candidates. Cosine similarity is multiplied by 20 before use.
- Distillation loss — KL divergence between the probability distribution of cosine similarities over its own 6 candidates and the probability distribution of teacher scores. The teacher temperature is 2.0.
For pairs, a contrastive loss is used with documents from other rows in the batch as negatives.
| Setting | Value |
|---|---|
| learning_rate | 2e-5 |
| num_train_epochs | 1 |
| per_device_train_batch_size | 64 |
| scale (multiplier for cosine similarity) | 20.0 |
| warmup_steps | 5% of total |
| weight_decay | 0.01 |
| Maximum input length during training | 512 tokens |
| Stored maximum input length for inference | 8,192 tokens |
| Mixed precision | bf16 |
| batch_sampler | NO_DUPLICATES |
| Gradient checkpointing | Enabled |
| Teacher temperature | 2.0 |
| Random seed | 42 |
Training took about 4 hours 27 minutes (5,923 steps) on a single NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB) GPU, with peak GPU memory usage of 22.1 GB.
4. License and Training Data Restrictions
The model weights are released under the MIT License. The base model, sbintuitions/modernbert-ja-310m, is also released under the MIT License.
The training data includes a Japanese translation of MS MARCO (mahiyama/mmarco-ja). MS MARCO is released for non-commercial research purposes, and its license for commercial use is not clear. If you use this model commercially, please check the MS MARCO terms of use at your own responsibility.
Part of the training data includes questions and answers collected from FAQs published by Japanese government agencies and local governments. This data is subject to the terms of use of each source and is not redistributed in this model's repository. If you use this model, please determine the suitability for your application at your own responsibility.
- Downloads last month
- 58
Model tree for mahiyama/Sazanami-Embed-310m
Base model
sbintuitions/modernbert-ja-310m