bekko-embedding-v1
Collection
Compact multilingual embedding models: 8M/25M-active-parameter encoders, their pretrained checkpoints, and a Transformer-free static embedding model. • 9 items • Updated • 7
bekko-embedding-v1-a8m-pt
bekko-embedding-v1-a8m-pt is the pretrained model used as the base for bekko-embedding-v1-a8m. It is a starting point for additional training on a downstream task, not a model intended for direct use.
Fine-tune this checkpoint for your target task before using it. It has not received the final supervised retrieval fine-tuning and should not be used directly as a general-purpose embedding or retrieval model. For general-purpose retrieval, use bekko-embedding-v1-a8m.
Model Details
| Item | Value |
|---|---|
| Model type | Pretrained Sentence Transformer dense embedding model |
| Architecture | mmBERT (ModernBERT-style) encoder, 4 layers, hidden size 384 |
| Base model | hotchpotch/mmBERT-L4H384-pruned |
| Original backbone | jhu-clsp/mmBERT-small |
| Active parameters | 7,671,168 |
| Total parameters | 105,975,168 |
| Embedding dimension | 384 |
| Supported truncate dimensions | 256, 128, 64 |
| Max sequence length | 8192 tokens |
| Pooling | Mean pooling |
| Similarity | Cosine similarity |
| Query/document prefixes | None |
The model was pretrained with multilingual text pairs using Matryoshka representation learning and quantization-aware training. The 4-layer backbone retains layers 0, 1, 2, and 18 from mmBERT-small.
Intended Use
Use this checkpoint as initialization for further training on a specific downstream task. Appropriate uses include:
- fine-tuning for retrieval, classification, reranking, or another target task,
- reproducing or adapting the bekko fine-tuning pipeline,
- pretrain-versus-fine-tuned comparisons and ablation studies.
License
MIT
- Downloads last month
- 109