--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: fill-mask datasets: - scilons/SciLaD-all-text-v1 tags: - scientific - scholarly - roberta - encoder - masked-lm - scientific-language-processing model-index: - name: SciLaD-M-roberta results: - task: type: token-classification name: Named Entity Recognition dataset: name: BC5CDR type: bc5cdr metrics: - type: f1 value: 96.54 name: F1 (macro) - task: type: token-classification name: Named Entity Recognition dataset: name: JNLPBA type: jnlpba metrics: - type: f1 value: 93.50 name: F1 (macro) - task: type: token-classification name: Named Entity Recognition dataset: name: NCBI-disease type: ncbi-disease metrics: - type: f1 value: 91.63 name: F1 (macro) - task: type: token-classification name: Named Entity Recognition dataset: name: SciERC type: scierc metrics: - type: f1 value: 42.49 name: F1 (macro) - task: type: token-classification name: PICO Extraction dataset: name: EBM-NLP type: ebm-nlp metrics: - type: f1 value: 78.47 name: F1 (macro) - task: type: text-classification name: Relation Classification dataset: name: ChemProt type: chemprot metrics: - type: f1 value: 82.64 name: F1 (micro) - task: type: text-classification name: Relation Classification dataset: name: SciERC type: scierc-rel metrics: - type: f1 value: 72.10 name: F1 (macro) - task: type: text-classification name: Citation Intent Classification dataset: name: ACL-ARC type: acl-arc metrics: - type: f1 value: 59.08 name: F1 (macro) - task: type: text-classification name: Paper Field Classification dataset: name: MAG type: mag metrics: - type: f1 value: 73.59 name: F1 (macro) - task: type: text-classification name: Citation Intent Classification dataset: name: SciCite type: scicite metrics: - type: f1 value: 80.75 name: F1 (macro) - task: type: token-classification name: Dependency Parsing dataset: name: GENIA type: genia metrics: - type: accuracy value: 53.23 name: LAS - type: accuracy value: 58.57 name: UAS --- # SciLaD-M (RoBERTa tokenizer) `SciLaD-M-roberta` is a RoBERTa-base encoder pre-trained **from scratch** on the English split of the [SciLaD](https://arxiv.org/abs/2512.11192) corpus of scientific publications, using the **original RoBERTa tokenizer** (`FacebookAI/roberta-base`). This variant isolates the effect of the training corpus from the effect of the vocabulary, and makes the model drop-in compatible with pipelines that already tokenize with RoBERTa. A sibling model, [`scilons/SciLaD-M-custom`](https://huggingface.co/scilons/SciLaD-M-custom), uses a tokenizer trained from scratch on SciLaD and has a slightly higher average benchmark score; see [Evaluation](#evaluation) to decide which variant fits your use case. ## Intended use - **Primary**: fine-tuning on scientific-text tasks — NER (biomedical, CS), PICO extraction, relation classification, citation-intent and field-of-research classification, dependency parsing. - **Also**: masked-token fill-in, contextual embeddings for retrieval or clustering of scientific documents, and drop-in replacement in existing RoBERTa-tokenized pipelines. - **Out of scope**: text generation (this is an encoder, not a generative LM), instruction following, non-English text, high-stakes clinical/diagnostic use without further validation. ## How to use ```python from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline tok = AutoTokenizer.from_pretrained("scilons/SciLaD-M-roberta") model = AutoModelForMaskedLM.from_pretrained("scilons/SciLaD-M-roberta") fill = pipeline("fill-mask", model=model, tokenizer=tok) fill(f"Transformer models achieve state-of-the-art {tok.mask_token} on many NLP benchmarks.") ``` For fine-tuning on downstream tasks, load the appropriate task head, e.g. `AutoModelForTokenClassification` for NER or `AutoModelForSequenceClassification` for classification: ```python from transformers import AutoModelForTokenClassification model = AutoModelForTokenClassification.from_pretrained( "scilons/SciLaD-M-roberta", num_labels=NUM_LABELS ) ``` ## Training data Pre-trained on the **English clean plain-text split of SciLaD**, published on the Hugging Face Hub as [`scilons/SciLaD-all-text-v1`](https://huggingface.co/datasets/scilons/SciLaD-all-text-v1): 10.99M scientific publications (68.7B tokens, avg. 6k tokens/doc) harvested from open-access sources via an entirely open-source pipeline: - **Unpaywall** snapshot (14 June 2023) → PDFs → TEI XML with [Grobid](https://github.com/kermitt2/grobid) - **arXiv** → LaTeX → TEI XML with LaTeXML - **PMC / PLOS** → JATS XML → TEI XML with Pub2TEI - Language filtering (weighted fastText ≥ 0.8 English), [Gopher](https://arxiv.org/abs/2112.11446) quality heuristics, and MinHash near-duplicate removal (5-grams, 112 hash functions, 75% Jaccard threshold) via [Datatrove](https://github.com/huggingface/datatrove). See the paper for full dataset statistics and the open pipeline. ## Training procedure - **Architecture**: RoBERTa-base — 12 Transformer layers, ~110M parameters. - **Objective**: Masked Language Modeling (MLM). - **Tokenizer**: the original `FacebookAI/roberta-base` byte-pair tokenizer (vocabulary size 50,265), reused unchanged. **Only the tokenizer is reused — the model weights are initialised randomly and trained entirely from scratch on SciLaD.** - **Batching**: batch size 256 × 2 gradient-accumulation steps → effective batch size 1024. - **Hardware**: 2 × NVIDIA H200 80GB. - **Data split**: 80 / 10 / 10 (train / validation / test) at document level. - **Hyperparameters**: otherwise as in the original RoBERTa configuration. ## Evaluation Fine-tuned on 11 scientific-NLP benchmarks (5 task families); F1 unless noted, averaged over 3 seeds. A compact view of **SciLaD-M-roberta vs. SciBERT and SciLaD-M-custom**: | Task | Dataset | **SciLaD-M-roberta** | SciLaD-M-custom | SciBERT | |---|---|---:|---:|---:| | NER | BC5CDR | 96.54 | **97.20** | 95.07 | | NER | JNLPBA | 93.50 | 93.89 | **94.19** | | NER | NCBI-disease | **91.63** | 91.48 | 87.09 | | NER | SciERC | 42.49 | 41.03 | **54.39** | | PICO | EBM-NLP | **78.47** | 77.66 | 78.14 | | REL | ChemProt (micro) | 82.64 | **83.83** | 82.83 | | REL | SciERC | 72.10 | **81.23** | 77.10 | | CLS | CitationIntent | 59.08 | **64.46** | 47.33 | | CLS | MAG | 73.59 | 73.01 | **74.61** | | CLS | SciCite | 80.75 | 83.96 | **85.49** | | DEP | GENIA (LAS) | 53.23 | **53.33** | 36.31 | | DEP | GENIA (UAS) | **58.57** | 57.88 | 41.68 | | **Avg.** | | **73.55** | 74.91 | 71.19 | This variant is the **highest-scoring of the compared models on NCBI-disease, EBM-NLP, and GENIA (UAS)**. Its overall average (73.55) is slightly below `SciLaD-M-custom` (74.91) but still ahead of SciBERT (71.19), RoBERTa (65.50), ModernBERT (65.51), and BERT (59.47). Full per-task numbers (including BERT, RoBERTa, and ModernBERT baselines with standard deviations) are in the paper. ### When to prefer this variant - You already have a tokenizer-dependent pipeline built around `FacebookAI/roberta-base` and want to swap in a scientific-domain model without retokenizing. - You need PICO or NCBI-disease NER performance specifically (this variant is strongest there). Otherwise, consider [`scilons/SciLaD-M-custom`](https://huggingface.co/scilons/SciLaD-M-custom), which has the higher overall average. ## Limitations and biases - **English only.** The pre-training corpus was filtered to English; performance on other languages is not guaranteed. - **Domain skew.** The open-access corpus is heavily biomedical / life-sciences, with smaller but meaningful coverage of physical sciences, CS, and social sciences. Expect stronger performance on biomedical text than on, e.g., pure-CS or humanities benchmarks (visible in the lower SciERC-NER score). - **Tokenizer mismatch.** The tokenizer was built for general-domain RoBERTa, so scientific terms are often split into more sub-word pieces than necessary. This is the motivation for `SciLaD-M-custom`. - **No alignment.** The model has not been instruction- or safety-tuned. It reflects the distribution of the scientific literature it was trained on, including any biases therein. - **Encoder only.** Not suitable for open-ended generation. ## License Released under the **Apache License 2.0**. ## Citation If you use this model, please cite the accompanying paper: ```bibtex @misc{foppiano2025scilad, title = {SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing}, author = {Foppiano, Luca and Takeshita, Sotaro and Ortiz Suarez, Pedro and Borisova, Ekaterina and Abu Ahmad, Raia and Ostendorff, Malte and Barth, Fabio and Moreno-Schneider, Julian and Rehm, Georg}, year = {2025}, eprint = {2512.11192}, archivePrefix= {arXiv}, url = {https://arxiv.org/abs/2512.11192} } ``` ## Acknowledgements Developed by researchers affiliated with DFKI, Humboldt-Universität zu Berlin, ScienciaLAB, Common Crawl Foundation, and the University of Mannheim. Supported by NFDI4DS (DFG project 460234259) and the EU Horizon Europe project HIVEMIND (No. 101189745). Compute provided by ACCESS / JetStream 2 (allocation CIS220172), TACC, and the University of Mannheim / DFKI.