distilbert-lora-finance-qa

A DistilBERT model fine-tuned on financial QA data using LoRA (Low-Rank Adaptation) — a parameter-efficient fine-tuning technique that trains only 0.22% of model parameters while achieving 99.50% Exact Match on financial question answering.

---

Model Description

This model is fine-tuned from distilbert-base-uncased-distilled-squad on two financial datasets using LoRA adapters targeting the query and value attention projections. The LoRA weights have been merged into the base model for clean, dependency-free inference.

Key highlight: Only 148,994 parameters were trained out of 66,513,412 total (0.22%) — demonstrating that large models can be efficiently adapted with minimal compute.

---

Training Details

Base Model

• distilbert-base-uncased-distilled-squad
• Already fine-tuned on SQuAD — strong QA starting point

Datasets

Dataset	Examples	Task
FiQA (explodinggradients/fiqa)	30	Financial QA
FinancialPhraseBank (FinanceMTEB/financial_phrasebank)	2264	Sentiment → QA
Total train	1835
Total eval	459

LoRA Configuration

Parameter	Value
Rank (r)	8
Alpha	16
Dropout	0.1
Target modules	q_lin, v_lin
Trainable params	148,994 (0.22%)
Total params	66,513,412

Training Arguments

Argument	Value
Epochs	3
Batch size	4
Gradient accumulation	8 (effective batch = 32)
Learning rate	3e-4
LR scheduler	cosine
Warmup ratio	0.1
Weight decay	0.01
Hardware	CPU
Training time	~1hr 43min
Optimizer	AdamW

---

Evaluation Results

Evaluated on 200 held-out examples using SQuAD-style Exact Match and F1:

Metric	Score
Exact Match (EM)	99.50%
F1 Score	99.50%
Eval Loss	0.0218

---

Ablation Study — LoRA Rank Comparison

Three training runs (500 steps each) with different LoRA ranks:

Rank	Trainable Params	Train Loss	Eval Loss	EM	F1
r=4	75,266 (0.11%)	0.0464	0.0225	99.50%	99.50%
r=8	148,994 (0.22%)	0.0378	0.0218	99.50%	99.50%
r=16	296,450 (0.45%)	0.0277	0.0202	99.50%	99.50%

Finding: All ranks achieve identical EM/F1, confirming that r=4 is the most efficient choice — same performance with 4x fewer parameters than r=16. The task (financial sentiment extraction) is simple enough that even the smallest adapter learns it perfectly.

---

How to Use

from transformers import pipeline

# Load directly from Hub — no PEFT needed!
qa = pipeline(
    "question-answering",
    model="Aditya020705/distilbert-lora-finance-qa"
)

# Financial sentiment extraction
result = qa(
    question="What is the financial sentiment of this statement?",
    context="positive. Apple reported record profits this quarter beating all expectations."
)
print(result["answer"])   # → positive
print(result["score"])    # → 0.9998

# Financial QA
result = qa(
    question="What caused the stock to drop?",
    context="The stock dropped after the company missed earnings by 30% and withdrew annual guidance."
)
print(result["answer"])   # → missed earnings by 30%

---

Training Loss Curve

Epoch	Train Loss
0.85	0.334
1.70	0.073
2.55	0.077
Final	0.1497

Loss dropped from 0.334 → 0.073 in the first epoch, showing rapid adaptation of the LoRA adapters to financial domain language.

---

Limitations

• Small dataset: Trained on only 1835 examples — may not generalize to all financial text types
• Extractive only: Finds spans in given context, cannot generate answers
• Sentiment format: For sentiment tasks, context must follow "[sentiment]. [sentence]" format
• Domain specific: Optimized for financial text — may degrade on general domain questions
• English only: Base model is English-only

---

Project Context

This model was built as part of a 21-day transformer project covering:

• Week 1: Transformer architecture from scratch
• Week 2: LoRA math + PEFT implementation
• Week 3: Full training run + ablation study ← this model
• Week 4: HuggingFace Hub publish + Gradio demo

Demo: Coming soon — Day 18-21 Gradio Space