Instructions to use ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF with sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF")

sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium."
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible embeddings server:
llama serve -hf ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF:Q4_K_M --embeddings

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible embeddings server:
llama serve -hf ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF:Q4_K_M --embeddings

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible embeddings server:
./llama-server -hf ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF:Q4_K_M --embeddings

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible embeddings server:
./build/bin/llama-server -hf ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF:Q4_K_M --embeddings

Llama new
LM Studio
Jan
Ollama
How to use ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF with Ollama:
```
ollama run hf.co/ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF with Docker Model Runner:
```
docker model run hf.co/ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF:Q4_K_M
```

Lemonade

How to use ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.BioClinical-ModernBERT-Embeddings-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

BioClinical ModernBERT Embeddings GGUF

Source model · HF release · Build hub

Pinned conversion of the Apache-2.0 BioClinical ModernBERT mean-pooling sentence embedding model. The native ModernBERT graph has 22 layers, 768 dimensions, local attention, and an 8,192-token native context window.

Formats

File	Status	Purpose
`bioclinical-modernbert-F16.gguf`	Published	Reference embeddings
`bioclinical-modernbert-Q8_0.gguf`	Published	Higher-quality compact embeddings
`bioclinical-modernbert-Q4_K_M.gguf`	Published	Smallest release format

Verified retrieval

Real llama-embedding output, --pooling mean --embd-normalize 2, 768-dimensional L2-normalised vectors, cosine similarity. Query: "Which drug class is used to treat type 2 diabetes?"

Rank	Q4_K_M	Q8_0	F16	Document
1	0.7165	0.7094	0.7058	Metformin is a first-line oral medication for type 2 diabetes mellitus.
2	0.4085	0.3903	0.3870	Insulin glargine is a long-acting basal insulin analogue for diabetes.
3	0.2887	0.2768	0.2751	Amoxicillin is a beta-lactam antibiotic used for bacterial infections.
4	0.1376	0.1267	0.1239	Aspirin irreversibly inhibits cyclooxygenase enzymes in platelets.
5	0.0714	0.0640	0.0658	The Airbus A320 is a narrow-body commercial airliner.
6	0.0270	0.0163	0.0177	The Great Wall of China was built across northern China.

All three quantizations produce the same ranking, and the off-domain sentences land at the bottom, so Q4_K_M is usable for retrieval. Scores are cosine values, not probabilities, and the margins are not calibrated confidence.

Validation

Twenty PubMedQA query/document pairs, 768-dimensional mean-pooled vectors, L2-normalized comparison:

Format	Mean cosine to source	Top-1 retrieval
F16	0.999998	0.95
Q8_0	0.999672	0.95
Q4_K_M	0.984172	0.95

The F16 GGUF matches the original SentenceTransformer embedding path. Retrieval agreement is measured against the 20-pair set, not a large benchmark.

Build

python -m pip install -r requirements-build.txt
python build_gguf.py --model-id NeuML/bioclinical-modernbert-base-embeddings

The builder pins llama.cpp commit 6b790a9c291b5d7af3312bbf9f0c558aa023b13e and the upstream model revision. It does not upload or overwrite this repository.

License

Apache-2.0. See the upstream model card and LICENSE for source attribution.

Downloads last month: 150

GGUF

Model size

0.1B params

Architecture

modern-bert

Hardware compatibility

4-bit

8-bit

16-bit

Model tree for ShayonSarker/BioClinical-ModernBERT-Embeddings-GGUF

Base model

answerdotai/ModernBERT-base

Finetuned

thomas-sounack/BioClinical-ModernBERT-base

Finetuned

NeuML/bioclinical-modernbert-base-embeddings

Quantized

(3)

this model