LFM2.5-VL-1.6B Japanese financial-statement extraction LoRA + model card

README.md

@@ -0,0 +1,93 @@
+---
+base_model: LiquidAI/LFM2.5-VL-1.6B
+library_name: peft
+pipeline_tag: image-text-to-text
+language:
+- ja
+tags:
+- lora
+- peft
+- vision-language
+- japanese
+- finance
+- document-extraction
+- on-device
+- liquid-ai
+- lfm2
+license: other
+license_name: lfm-open-license-1.0
+license_link: https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B
+---
+
+# LFM2.5-VL-1.6B · Japanese Financial-Statement Extraction (LoRA)
+
+A **LoRA adapter** for [`LiquidAI/LFM2.5-VL-1.6B`](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B)
+that teaches the on-device vision-language model to read **Japanese financial statements
+(決算書 / 決算短信)** and return the key figures as structured JSON — entirely on-device.
+
+Built for an on-premise bank credit-underwriting tool where customer financials can't leave the
+machine (APPI / FISC). The stock 1.6B model reads clean statements but fails on dense, real
+multi-column 決算短信; this adapter fixes that.
+
+## Results (held-out real 短信, never seen in training)
+Field accuracy on 7 core figures across 3 real filings (明治HD / エディオン / カプコン),
+graded against EDINET-exact gold:
+
+| | Field accuracy |
+|---|---|
+| Base `LFM2.5-VL-1.6B` | **14.3%** |
+| **+ this LoRA** | **85.7%** |
+
+Trajectory across training iterations: **14.3 → 28.6 → 38.1 → 52.4 → 76.2 → 85.7 %** (≈6×).
+Residual errors are single-digit slips in 6–7-digit numbers (a precision limit of a 1.6B VL),
+not structural failures.
+
+## What it extracts
+Per statement, as JSON (¥ millions): 決算期, 証券コード, 売上高, 営業利益, 経常利益, 当期純利益,
+総資産, 純資産, 有利子負債, 現預金, 前期売上高, 前期営業利益, 営業CF, 投資CF, 財務CF.
+
+## Usage
+```python
+import torch
+from transformers import AutoProcessor, AutoModelForImageTextToText
+from peft import PeftModel
+from PIL import Image
+
+BASE = "LiquidAI/LFM2.5-VL-1.6B"
+ADAPTER = "DjTechzz/LFM2.5-VL-1.6B-JP-Financial-Extract-LoRA"
+
+proc = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
+model = AutoModelForImageTextToText.from_pretrained(
+    BASE, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda")
+model = PeftModel.from_pretrained(model, ADAPTER).eval()
+
+prompt = ("あなたは財務書類の読み取り専用エンジンです。添付の決算書画像から"
+          "売上高・営業利益・経常利益・当期純利益・総資産・純資産・現預金 等を読み取り、"
+          "JSONのみを出力。金額は百万円単位の整数。前期/当期が並ぶ場合は当期を採用。")
+img = Image.open("kessan.png").convert("RGB")
+msgs = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]
+text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
+inp = proc(text=[text], images=[[img]], return_tensors="pt").to(model.device)
+out = model.generate(**inp, max_new_tokens=512, do_sample=False)
+print(proc.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+Runs on-device (works on an AMD Ryzen AI / NVIDIA laptop). Pair the JSON with a deterministic
+ratio/rating engine for an auditable credit memo.
+
+## Training
+- **Method:** LoRA (rank 16, α 32; q/k/v/o + gate/up/down proj), bf16, grad-checkpointing, lr 1e-4.
+- **Stack:** `transformers` + `peft` (no unsloth/triton), CUDA. ~30 min on an RTX PRO 5000.
+- **Data (no manual labels):** synthetic statements rendered from real **EDINET** financials (exact
+  labels, incl. a realistic 決算短信 multi-column template) + **Qwen2.5-VL-7B distillation** on real
+  IR 決算短信; real labels later replaced with **EDINET-exact** values to remove teacher digit-noise.
+- **Tracking:** Weights & Biases.
+
+## Intended use & limitations
+For on-device extraction of standardized Japanese financial statements to assist credit analysis.
+Outputs are a drafting aid — **not** a lending decision; verify figures before use. Occasional
+single-digit OCR slips on very large numbers. Best on summary-style 短信 / 決算書 layouts.
+
+## License
+This is a derivative LoRA adapter; use is subject to the base model's license
+([`LiquidAI/LFM2.5-VL-1.6B`](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B)). Built during the
+Liquid AI × WAY × AMD hackathon (Tokyo, 2026).

adapter_config.json

@@ -0,0 +1,51 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Lfm2VlForConditionalGeneration",
+    "parent_library": "transformers.models.lfm2_vl.modeling_lfm2_vl"
+  },
+  "base_model_name_or_path": "LiquidAI/LFM2.5-VL-1.6B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "k_proj",
+    "up_proj",
+    "down_proj",
+    "v_proj",
+    "o_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": null,
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88dab80e2b9d2fa7f3d4a2e3f609a0040d3c0e3fbe0e19f34fafb2e13d4099b7
+size 15513232

chat_template.jinja

@@ -0,0 +1,65 @@
+{{- bos_token -}}
+{%- set keep_past_thinking = keep_past_thinking | default(false) -%}
+{%- set ns = namespace(system_prompt="") -%}
+{%- if messages[0]["role"] == "system" -%}
+    {%- set sys_content = messages[0]["content"] -%}
+    {%- if sys_content is not string -%}
+        {%- for item in sys_content -%}
+            {%- if item["type"] == "text" -%}
+                {%- set ns.system_prompt = ns.system_prompt + item["text"] -%}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {%- set ns.system_prompt = sys_content -%}
+    {%- endif -%}
+    {%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+    {%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
+    {%- for tool in tools -%}
+        {%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+        {%- endif -%}
+        {%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns.system_prompt = ns.system_prompt + "]" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- set ns.last_assistant_index = -1 -%}
+{%- for message in messages -%}
+    {%- if message["role"] == "assistant" -%}
+        {%- set ns.last_assistant_index = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+{%- for message in messages -%}
+    {{- "<|im_start|>" + message["role"] + "\n" -}}
+    {%- if message["content"] is not string -%}
+        {%- set ns.content = "" -%}
+        {%- for item in message["content"] -%}
+            {%- if item["type"] == "image" -%}
+                {%- set ns.content = ns.content + "<image>" -%}
+            {%- elif item["type"] == "text" -%}
+                {%- set ns.content = ns.content + item["text"] -%}
+            {%- else -%}
+                {%- set ns.content = ns.content + item | tojson -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {%- set content = ns.content -%}
+    {%- else -%}
+        {%- set content = message["content"] -%}
+    {%- endif -%}
+    {%- if message["role"] == "assistant" and not keep_past_thinking and loop.index0 != ns.last_assistant_index -%}
+        {%- if "</think>" in content -%}
+            {%- set content = content.split("</think>")[-1] | trim -%}
+        {%- endif -%}
+    {%- endif -%}
+    {{- content + "<|im_end|>\n" -}}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{- "<|im_start|>assistant\n" -}}
+{%- endif -%}

processor_config.json

@@ -0,0 +1,39 @@
+{
+  "image_processor": {
+    "data_format": "channels_first",
+    "do_image_splitting": true,
+    "do_normalize": true,
+    "do_pad": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "downsample_factor": 2,
+    "encoder_patch_size": 16,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Lfm2VlImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_image_tokens": 256,
+    "max_num_patches": 1024,
+    "max_pixels_tolerance": 2.0,
+    "max_tiles": 10,
+    "min_image_tokens": 64,
+    "min_tiles": 2,
+    "resample": 2,
+    "rescale_factor": 0.00392156862745098,
+    "return_row_col_info": true,
+    "size": {
+      "height": 512,
+      "width": 512
+    },
+    "tile_size": 512,
+    "use_thumbnail": true
+  },
+  "processor_class": "Lfm2VlProcessor"
+}

tokenizer_config.json

@@ -0,0 +1,27 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|im_end|>",
+  "image_end_token": "<|image_end|>",
+  "image_start_token": "<|image_start|>",
+  "image_thumbnail": "<|img_thumbnail|>",
+  "image_token": "<image>",
+  "is_local": false,
+  "legacy": false,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "image_end_token": "<|image_end|>",
+    "image_start_token": "<|image_start|>",
+    "image_token": "<image>"
+  },
+  "pad_token": "<|pad|>",
+  "processor_class": "Lfm2VlProcessor",
+  "return_token_type_ids": false,
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false,
+  "use_fast": true
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f14edc01fdacdeda89dcf598c1a06d61c54a242276b0cfcd653c1b945462a25
+size 5265