CamemBERT-bio-v2-base

CamemBERT-bio-v2-base is a French biomedical encoder for token-level tasks. It is obtained by continuing the pretraining of camembertv2-base on biomedical text. On our named entity recognition benchmark it reaches 65.8 exact-F1, among the best results we measured for a French biomedical encoder of this size. For long-document tasks, see ModernCamemBERT-bio-v2-large.

Authors

Rian Touchent and Eric de la Clergerie, Sorbonne Université, Inria Paris.

Overview

This model targets clinical and biomedical entity recognition in French. Starting from camembertv2-base, we first run a causal language modeling detour on clinical text, then a decay phase that masks whole entity spans and gives more weight to medical terms. The detour deepens the domain adaptation, and the entity-focused masking sharpens the token representations that entity recognition relies on.

Architecture	RoBERTa (CamemBERTv2)
Parameters	111M
Context length	1,024 tokens
Language	French
Base model	almanach/camembertv2-base

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "rntc/CamemBERT-bio-v2-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

text = "Le patient présente une <mask> aiguë du myocarde."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits
i = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, i].argmax(-1)))

Training

The model is initialized from camembertv2-base. We first apply a causal language modeling detour on the MC-Bio clinical corpus, which produces lasting changes in the early layers and improves domain adaptation. We then run a decay phase with whole-entity-span masking, where medical surface forms are masked as full spans and sampled more often than ordinary words. This keeps the masked language modeling objective throughout.

Evaluation

Per-task results on the short named entity recognition core of our French biomedical benchmark, scored with exact-F1. Our models are averaged over nine seeds; the external baselines are currently averaged over three seeds.

Model	EMEA	Medline	E3C	Avg
DoctoBERT	72.0	66.5	61.5	66.7
CamemBERT-bio-v2	71.9	64.7	60.9	65.8
CamemBERT-bio	70.6	65.1	61.5	65.7
DoctoModernBERT	68.7	65.4	60.7	64.9
DrBERT	69.0	62.9	58.5	63.4
ModernCamemBERT-bio-v2-large	67.8	63.4	58.7	63.3
ModernCamemBERT-bio-base	67.6	60.7	58.7	62.3
ModernCamemBERT-bio-v2-base	64.0	59.6	58.5	60.7

CamemBERT-bio-v2 comes second on the average, close behind doctobert and ahead of DoctoModernBERT. Most of the remaining gap comes from Medline, the scientific-title subset.

Environmental impact

The biomedical continued-pretraining reported here ran for about 1 GPU-hour on H100, on the Jean Zay cluster (GENCI-IDRIS) in France. We estimate roughly 0.03 kg CO2eq for this phase. This figure covers the continued-pretraining step only and does not include the pretraining of the base model.

License

MIT

Citation

@inproceedings{touchent:hal-05697506,
  TITLE = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
  AUTHOR = {Touchent, Rian and de la Clergerie, {\'E}ric},
  URL = {https://hal.science/hal-05697506},
  BOOKTITLE = {{Proceedings of Knowledge Graphs and Large Language Models Workshop}},
  ADDRESS = {Palma de Mallorca, Spain},
  YEAR = {2026},
  MONTH = May,
  PDF = {https://hal.science/hal-05697506v1/file/main.pdf},
  HAL_ID = {hal-05697506},
  HAL_VERSION = {v1},
}

@misc{touchent2026causallanguagemodelingdetour,
  title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
  author={Rian Touchent and Eric de la Clergerie},
  year={2026},
  eprint={2605.12438},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2605.12438},
}