bsye/byosm-doctype

Document type

Document type classifies business document pages into the 16 RVL-CDIP classes, such as letter, invoice, form and scientific report. It runs on CPU as an int8 ONNX export with a one-file Python runtime.

At 109.5M parameters this model is not smaller than the best rival, microsoft/dit-base-finetuned-rvlcdip (85.8M), because it is built on that rival's own checkpoint. It scores 0.948 accuracy against the rival's 0.94.

Table of Contents

• Overview
• How It Works
• Architecture and Training
  • Base Model
  • Training Data
  • Pretraining and Refinement
  • Quantization and Export)
• Benchmarks
• Usage
• Deployment Requirements
• Limitations
• License
• Resources
• Reproduce

Overview

Document pipelines often need to route incoming pages by type before extraction, filing or review. The inputs are scanned or born-digital business documents, mostly single page, typically grayscale scans of about 1000px.

A single-task specialist does one thing and runs locally on CPU, without a hosted service. Use it when your pages fit the RVL-CDIP taxonomy and are in English.

The model predicts one of these labels:

Label	Description
advertisement	promotional material: ads, product promotions, flyers, posters
budget	budget tables, financial plans, cost breakdowns and spending summaries
email	printed e-mail messages (From/To/Subject headers)
file_folder	file folder covers, tabs and labels
form	fill-in forms with fields, boxes and checkboxes
handwritten	documents that are mostly handwritten
invoice	bills and invoices: amounts due, line items, totals
letter	formal correspondence: letterhead, salutation, signature
memo	internal memoranda (MEMO / TO / FROM / RE headers)
news_article	newspaper or magazine articles and clippings
presentation	slides and presentation pages
questionnaire	surveys and questionnaires with questions to answer
resume	resumes / CVs and biographical sketches
scientific_publication	published scientific papers and journal pages
scientific_report	scientific / technical reports, lab reports
specification	product or technical specifications, spec sheets

How It Works

1. Input: a document page as an image or PDF. For scanned pages, the text is extracted with OCR. The text input is limited to 256 tokens.
2. Prediction: the model uses a fused route that combines the page image with the document text, then scores all 16 classes.
3. Decision threshold: none. This is a single-label, 16-way task.
4. Output: the label with the highest score.

Architecture and Training

Base Model

The image backbone is microsoft/dit-base-finetuned-rvlcdip, which is the best rival in the benchmark, and the image features come from its pooler output. Earlier experiments used facebook/dinov2-small and then microsoft/dit-base. DiT-base is self-supervised with masked image modelling on 42M unlabelled document scans and never saw RVL-CDIP labels. The final step switched to the RVL-CDIP fine-tuned checkpoint.

Training Data

• chainyo/rvl-cdip: 29112 labeled training rows, drawn from its validation split
• Unlabeled train inputs: 29112 rows
• Total training rows: 34935
• Dev rows: 3881
• Test rows: 3200, held out

Pretraining and Refinement

• Pretraining steps: 0
• Epochs: 13
• Refinement: one round, starting from 34935 rows, which reached a dev score of 0.942. No refinement patterns were applied.
• Lineage: per-label data went from 200 to 800 and then to 2426 (all of it), the backbone moved to a document-domain encoder, and training was extended to 13 epochs. The variant built on the DiT-base backbone scored 0.906 on the test set before the final switch to the rival's fine-tuned checkpoint.

Quantization and Export

• Format: ONNX, in a single folder with a one-file Python runtime
• Precision: int8
• Size: 113.2 MB

Benchmarks

This model and microsoft/dit-base-finetuned-rvlcdip are compared on accuracy over 3200 held-out test items from chainyo/rvl-cdip. The comparison also reports the share of each label that each model caught. The 95% confidence interval for this model is shown next to its score.

Results, measured by code

Every number in this section was computed by BYOSM's scorer on frozen splits; none was written by an LLM. This model and its rivals were scored by the same code, each with its decision threshold calibrated on the same calib split. Bold = this model.

model	params	builtin accuracy	rvl accuracy (primary)	builtin ROC-AUC	rvl ROC-AUC
v2-ditft (this model)	109.55M	–	0.9481	–	–
hf:microsoft/dit-base-finetuned-rvlcdip (baseline)	85.82M	0.9271	0.9400	–	–
v1 (BYOSM experiment)	109.55M	–	0.9184	–	–
r2 (BYOSM experiment)	109.55M	–	0.9050	–	–
fused-dit (BYOSM experiment)	109.55M	0.8792	0.8838	–	–
fused (BYOSM experiment)	45.70M	0.8063	0.8009	–	–

This model, per suite

suite	dataset	test items	accuracy	95% CI	precision	recall	ROC-AUC	avg precision	decision threshold
rvl	chainyo/rvl-cdip	3200	0.9481	0.941-0.956	–	–	–	–	–

Model details

task	document / classify
labels	advertisement, budget, email, file_folder, form, handwritten, invoice, letter, memo, news_article, presentation, questionnaire, resume, scientific_publication, scientific_report, specification
parameters	109,546,385 (shipped as int8 ONNX, 113 MB)
backbone	text hf:sentence-transformers/all-MiniLM-L6-v2 + visual hf:microsoft/dit-base-finetuned-rvlcdip, route fused · embedding dim 256
objective	accuracy, primary suite rvl
decision policy	threshold, threshold – (calibrated on rvl's calib split)

This model scores 0.948 (95% CI 0.941 to 0.956) against 0.94 for the rival, which sits just below the lower bound of the interval. The two models are close: they tie on news_article, questionnaire, resume, scientific_publication and scientific_report, the rival is ahead on advertisement (0.946 vs 0.941), and both models are weakest on scientific_report (0.876).

Usage

Load the exported folder with the bundled Python runtime and pass it a page image or PDF.

pip install huggingface_hub numpy onnxruntime tokenizers pillow pypdfium2 rapidocr_onnxruntime

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("bsye/byosm-doctype")
sys.path.insert(0, path)
from byosm_rt import load

m = load(path)
print(m.predict("invoice.pdf"))       # PDF or page image; scans are OCR'd
emb = m.encode(...)                # the learned representation (JEPA-style), for probes / retrieval / other heads

Local HTTP server

python byosm_rt.py serve . --port 8800
# GET /v1/models · POST /v1/doctype/predict · POST /v1/doctype/encode

Deployment Requirements

• CPU only, no GPU required
• Size: 113.2 MB (int8 ONNX), 109.5M parameters
• Latency: about 43.3 ms per item on CPU
• Dependencies: numpy, onnxruntime, tokenizers, pillow, pypdfium2, rapidocr_onnxruntime (OCR for scanned pages)

Limitations

• Evaluated only on rvl (3200 test items). Small test suites have wide confidence intervals.
• English only.
• Covers only the 16 RVL-CDIP classes. Pages outside this taxonomy are still assigned one of them.
• Built on the rival's RVL-CDIP fine-tuned checkpoint, so it is larger than that rival (109.5M vs 85.8M parameters). Its accuracy lead is small.
• Lowest per-label recall is on scientific_report (0.876) and form (0.905).
• Scanned pages depend on OCR quality for the text input, which is limited to 256 tokens.

License

cc-by-nc-4.0

Resources

• BYOSM on GitHub
• chainyo/rvl-cdip (training and benchmark data)
• microsoft/dit-base-finetuned-rvlcdip (rival and base checkpoint)
• microsoft/dit-base (earlier backbone)

Reproduce

git clone https://github.com/bsye/BYOSM && cd BYOSM && pip install -e .
# put spec.yaml from this repo at specialists/doctype/spec.yaml, then:
byosm build doctype && byosm board doctype && byosm export doctype

Task definition

From spec.yaml, shipped in this repo.

Task. 16-way document type classification (RVL-CDIP classes)

Inputs. scanned or born-digital business documents, mostly single page, grayscale scans ~1000px

Labels.

• advertisement: promotional material: ads, product promotions, flyers, posters
• budget: budget tables, financial plans, cost breakdowns and spending summaries
• email: printed e-mail messages (From/To/Subject headers)
• file_folder: file folder covers, tabs and labels
• form: fill-in forms with fields, boxes and checkboxes
• handwritten: documents that are mostly handwritten
• invoice: bills and invoices: amounts due, line items, totals
• letter: formal correspondence: letterhead, salutation, signature
• memo: internal memoranda (MEMO / TO / FROM / RE headers)
• news_article: newspaper or magazine articles and clippings
• presentation: slides and presentation pages
• questionnaire: surveys and questionnaires with questions to answer
• resume: resumes / CVs and biographical sketches
• scientific_publication: published scientific papers and journal pages
• scientific_report: scientific / technical reports, lab reports
• specification: product or technical specifications, spec sheets

---

Prose by claude-opus-5-5 from the facts above; every number is checked against them.