BYOSM publish doctype/v2-ditft

README.md

@@ -30,15 +30,19 @@ base_model_relation: finetune
 ---
 # Document type
 
-**bsye/byosm-doctype** is a small specialist model that sorts business documents into the **16 RVL-CDIP document classes**. It runs on CPU as an **int8 ONNX** export and reads both the page image and its OCR text.
+**Document type** classifies business document pages into the **16 RVL-CDIP classes**, such as letter, invoice, form and scientific report. It runs on CPU as an **int8 ONNX** export with a one-file Python runtime.
 
-On a held-out set of 3200 RVL-CDIP pages it reaches **0.948 accuracy** (95% CI 0.941 to 0.956), compared with 0.94 for microsoft/dit-base-finetuned-rvlcdip.
+At **109.5M** parameters this model is **not smaller** than the best rival, **microsoft/dit-base-finetuned-rvlcdip** (**85.8M**), because it is **built on that rival's own checkpoint**. It scores **0.948** accuracy against the rival's **0.94**.
 
 ## Table of Contents
 
 - [Overview](#overview)
 - [How It Works](#how-it-works)
 - [Architecture and Training](#architecture-and-training)
+  - [Base Model](#base-model)
+  - [Training Data](#training-data)
+  - [Pretraining and Refinement](#pretraining-and-refinement)
+  - [Quantization and Export)](#quantization-and-export)
 - [Benchmarks](#benchmarks)
 - [Usage](#usage)
 - [Deployment Requirements](#deployment-requirements)
@@ -49,22 +53,11 @@ On a held-out set of 3200 RVL-CDIP pages it reaches **0.948 accuracy** (95% CI 0
 
 ## Overview
 
-Document pipelines often need to know what kind of document they are looking at before they process it: an invoice goes to accounts, a resume goes to hiring, a form goes to data extraction. This model gives that answer as a single **document type** label for each page.
+Document pipelines often need to route incoming pages by type before extraction, filing or review. The inputs are **scanned or born-digital business documents**, mostly single page, typically grayscale scans of about 1000px.
 
-A **small specialist** is a good fit here because the label set is fixed and the inputs are mostly alike. It does not need a general-purpose model, it runs locally on CPU, and its cost per document stays predictable.
+A single-task specialist does one thing and runs locally on CPU, without a hosted service. Use it when your pages fit the RVL-CDIP taxonomy and are in **English**.
 
-Use it when:
-
-- inputs are **scanned or born-digital business documents**, mostly single page (typically grayscale scans of around 1000px)
-- the text is in **English**
-- the RVL-CDIP class set fits your routing needs
-
-## How It Works
-
-1. **Input**: a document page as an image or PDF. The runtime renders PDFs and runs **OCR** on scanned pages. OCR text is cut off at **256 tokens**.
-2. **Prediction**: the model combines image features with the OCR text (the **fused** route) and scores all 16 classes.
-3. **Decision threshold**: none is set. The model picks the class with the highest score.
-4. **Output**: one label from the set below.
+The model predicts one of these labels:
 
 | Label | Description |
 |---|---|
@@ -85,43 +78,43 @@ Use it when:
 | `scientific_report` | scientific / technical reports, lab reports |
 | `specification` | product or technical specifications, spec sheets |
 
-## Architecture and Training
+## How It Works
 
-### Base Model
+1. **Input**: a document page as an image or PDF. For scanned pages, the text is extracted with OCR. The text input is limited to **256 tokens**.
+2. **Prediction**: the model uses a **fused** route that combines the page image with the document text, then scores all 16 classes.
+3. **Decision threshold**: none. This is a single-label, 16-way task.
+4. **Output**: the label with the highest score.
 
-The image backbone is **microsoft/dit-base-finetuned-rvlcdip**, with **pooler** output used as the image representation. That checkpoint builds on **microsoft/dit-base**, which was pretrained with self-supervised masked image modelling on 42M unlabelled document scans. The full model has **109.5M parameters**.
+## Architecture and Training
 
-The backbone was chosen over a few iterations:
+### Base Model
 
-- Early experiments used `facebook/dinov2-small`, which was pretrained on a different domain.
-- That backbone was then replaced with `microsoft/dit-base`, a document-domain encoder.
-- The final experiment (**v2-ditft**) switched to the RVL-CDIP fine-tuned DiT checkpoint. This is the same model used as the rival in the benchmark below.
+The image backbone is **microsoft/dit-base-finetuned-rvlcdip**, which is the best rival in the benchmark, and the image features come from its **pooler** output. Earlier experiments used `facebook/dinov2-small` and then **microsoft/dit-base**. DiT-base is self-supervised with masked image modelling on 42M unlabelled document scans and never saw RVL-CDIP labels. The final step switched to the RVL-CDIP fine-tuned checkpoint.
 
 ### Training Data
 
-- **hf:chainyo/rvl-cdip**: 29112 labelled training rows
-- **train-inputs** (unlabelled): 29112 rows
-- Total training rows: **34935**
-- Dev rows: **3881**
-- Held-out test rows: **3200**
-
-The amount of data per label went up over several experiments, ending with all available rows for the rarest label in the source split.
+- **chainyo/rvl-cdip**: **29112** labeled training rows, drawn from its validation split
+- **Unlabeled train inputs**: **29112** rows
+- **Total training rows**: **34935**
+- **Dev rows**: **3881**
+- **Test rows**: **3200**, held out
 
 ### Pretraining and Refinement
 
-- **Pretraining**: none (0 pretraining steps).
-- **Training**: 13 epochs.
-- **Refinement**: one round, starting from 34935 rows, which reached a dev accuracy of **0.942**. No refinement patterns were applied.
+- **Pretraining steps**: 0
+- **Epochs**: 13
+- **Refinement**: one round, starting from **34935** rows, which reached a dev score of **0.942**. No refinement patterns were applied.
+- **Lineage**: per-label data went from 200 to 800 and then to **2426** (all of it), the backbone moved to a document-domain encoder, and training was extended to 13 epochs. The variant built on the DiT-base backbone scored **0.906** on the test set before the final switch to the rival's fine-tuned checkpoint.
 
 ### Quantization and Export
 
-- Format: **ONNX**, delivered as one folder with a one-file Python runtime
-- Precision: **int8**
-- Size: **113.2 MB**
+- **Format**: ONNX, in a single folder with a one-file Python runtime
+- **Precision**: int8
+- **Size**: **113.2 MB**
 
 ## Benchmarks
 
-This model and microsoft/dit-base-finetuned-rvlcdip were both scored on the same 3200 held-out rows from **chainyo/rvl-cdip**. Overall performance is reported as **accuracy**, and the per-label figures show the share of each class that was correctly caught (per-label recall). The confidence interval applies to this model's overall score.
+This model and **microsoft/dit-base-finetuned-rvlcdip** are compared on **accuracy** over **3200** held-out test items from **chainyo/rvl-cdip**. The comparison also reports the share of each label that each model caught. The 95% confidence interval for this model is shown next to its score.
 
 ### Results, measured by code
 
@@ -155,13 +148,13 @@ and its rivals were scored by the same code, each with its decision threshold ca
 | objective | accuracy, primary suite `rvl` |
 | decision policy | `threshold`, threshold – (calibrated on `rvl`'s calib split) |
 
-The model is slightly ahead overall and on most labels, but the margin is small: the rival's 0.94 falls just below the lower bound of this model's confidence interval, and the rival does better on `advertisement` (0.946 vs 0.941). The two models tie on `news_article`, `questionnaire`, `resume`, `scientific_publication` and `scientific_report`, and both have their lowest score on `scientific_report` (0.876), so the improvement over the backbone this model was built from is modest.
+This model scores **0.948** (95% CI **0.941** to **0.956**) against **0.94** for the rival, which sits just below the lower bound of the interval. The two models are close: they tie on `news_article`, `questionnaire`, `resume`, `scientific_publication` and `scientific_report`, the rival is ahead on `advertisement` (**0.946** vs **0.941**), and both models are weakest on `scientific_report` (**0.876**).
 
 
 
 ## Usage
 
-The bundled one-file Python runtime loads the ONNX model and returns a document type for each input page.
+Load the exported folder with the bundled Python runtime and pass it a page image or PDF.
 
 ```bash
 pip install huggingface_hub numpy onnxruntime tokenizers pillow pypdfium2 rapidocr_onnxruntime
@@ -188,38 +181,30 @@ python byosm_rt.py serve . --port 8800
 
 ## Deployment Requirements
 
-- **CPU only**, no GPU needed
-- Size: **113.2 MB** (int8 ONNX), 109.5M parameters
-- Latency: **43.3 ms** per document on CPU
-- Dependencies:
-  - `numpy`
-  - `onnxruntime`
-  - `tokenizers`
-  - `pillow`
-  - `pypdfium2`
-  - `rapidocr_onnxruntime` (OCR for scanned pages)
+- **CPU only**, no GPU required
+- **Size**: 113.2 MB (int8 ONNX), 109.5M parameters
+- **Latency**: about **43.3 ms** per item on CPU
+- **Dependencies**: `numpy`, `onnxruntime`, `tokenizers`, `pillow`, `pypdfium2`, `rapidocr_onnxruntime` (OCR for scanned pages)
 
 ## Limitations
 
-- Evaluated only on rvl (3200 test items). Small test suites have wide confidence intervals.
-- The backbone is **microsoft/dit-base-finetuned-rvlcdip**, the same model used as the rival. The benchmark therefore shows what additional training adds to that model, not how two independent models compare.
+- Evaluated only on rvl (**3200** test items). Small test suites have wide confidence intervals.
 - English only.
-- Limited to the 16 RVL-CDIP classes. Documents outside these classes still get one of the 16 labels.
-- Inputs are expected to be mostly single page business documents. Multi-page or very different layouts have not been evaluated.
-- OCR text is cut off at 256 tokens, so text-heavy pages are only partly read.
-- Recall is lowest on `scientific_report` (0.876) and `form` (0.905).
-- No decision threshold is set, so the model has no built-in way to abstain on low-confidence pages.
+- Covers only the 16 RVL-CDIP classes. Pages outside this taxonomy are still assigned one of them.
+- Built on the rival's RVL-CDIP fine-tuned checkpoint, so it is larger than that rival (**109.5M** vs **85.8M** parameters). Its accuracy lead is small.
+- Lowest per-label recall is on `scientific_report` (**0.876**) and `form` (**0.905**).
+- Scanned pages depend on OCR quality for the text input, which is limited to 256 tokens.
 
 ## License
 
-cc-by-nc-4.0 (non-commercial use only).
+cc-by-nc-4.0
 
 ## Resources
 
-- BYOSM on GitHub: [bsye/BYOSM](https://github.com/bsye/BYOSM)
-- Dataset: [chainyo/rvl-cdip](https://huggingface.co/datasets/chainyo/rvl-cdip)
-- Rival and backbone: [microsoft/dit-base-finetuned-rvlcdip](https://huggingface.co/microsoft/dit-base-finetuned-rvlcdip)
-- Backbone origin: [microsoft/dit-base](https://huggingface.co/microsoft/dit-base)
+- [BYOSM on GitHub](https://github.com/bsye/BYOSM)
+- [chainyo/rvl-cdip](https://huggingface.co/datasets/chainyo/rvl-cdip) (training and benchmark data)
+- [microsoft/dit-base-finetuned-rvlcdip](https://huggingface.co/microsoft/dit-base-finetuned-rvlcdip) (rival and base checkpoint)
+- [microsoft/dit-base](https://huggingface.co/microsoft/dit-base) (earlier backbone)
 
 ## Reproduce