LFM3-Omni Dual-Head Architecture

Unified System 1 (Fast Decision) & System 2 (Multimodal Generation) on a Shared Backbone

License: Apache 2.0 Base Models Library

This repository introduces the first unified Dual-Head architecture combining:

  1. System 1 (Decision Model): Fast, non-autoregressive decision-making based on LiquidAI/d1-Omni-2B (zero output tokens, single-pass classification/routing).
  2. System 2 (Generative Model): Autoregressive text and speech generation based on LiquidAI/LFM3-Omni-2B.

πŸ’‘ Why Dual-Head?

Previously, developers needing both instant event triage/routing and conversational abilities had to load two separate 2.2B models into GPU memory, consuming 8–10 GB of VRAM.

Because both models share the same Tri-hybrid LFM3-Omni trunk (gated short convolutions, sliding-window attention, and global attention), this architecture unifies them into a single object:

  • ⚑ ~50% VRAM Savings: Uses only ~4.5 GB VRAM (or ~1.8 GB in 4-bit quantization).
  • 🧠 Zero Weight Duplication: Shares projection tensors in memory directly using memory views.
  • 🎯 Dual Dispatch: Call .system_one(...) for sub-50ms decisions, and .generate(...) for full chat and multimodal responses.

πŸ› Architecture Diagram

                        [ Input: Text / Image / Audio ]
                                       β”‚
                                       β–Ό
             β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
             β”‚       Shared LFM3-Omni Backbone (~2.1B)         β”‚
             β”‚   (Tri-hybrid: Gated Conv + Sliding Attention)   β”‚
             β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                      β”‚
                 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                 β–Ό                                         β–Ό
   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”             β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
   β”‚       Decision Head       β”‚             β”‚       Language Head       β”‚
   β”‚      (from d1-Omni)       β”‚             β”‚     (from LFM3-Omni)      β”‚
   β”‚  Call via .system_one()   β”‚             β”‚   Call via .generate()    β”‚
   β”‚  Output: Typed Decisions  β”‚             β”‚   Output: Text & Speech   β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜             β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸš€ Quickstart & Usage

1. Installation

pip install -U "transformers>=4.40" torch pillow numpy huggingface_hub

2. Loading the Model

import torch
from transformers import AutoModel

device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    "JonusNattapong/lfm3-omni-dualhead",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
    device_map="auto" if device == "cuda" else None
)

3. System 1: Fast Calibrated Decision (Single Forward Pass)

# Triage customer intent & urgency in sub-50ms (0 output tokens)
state_text = "Urgent: payment webhook failed for order #1234"

questions = {
    "intent": {
        "type": "choice",
        "choices": ["payment_issue", "account_inquiry", "general_support"]
    },
    "is_urgent": {
        "type": "boolean"
    }
}

result = model.system_one(state=state_text, questions=questions)
print(result)
# Output: {'intent': 'payment_issue', 'is_urgent': True}

4. Multimodal Image Decision (224x224)

from PIL import Image

image = Image.open("sample.jpg").resize((224, 224))
questions = {
    "has_receipt": {"type": "boolean"},
    "quality_score": {"type": "number", "range": [1, 10], "step": 1}
}

image_decision = model.system_one(media={"image": image}, questions=questions)
print(image_decision)

5. Multimodal Audio Event Routing (16kHz Audio)

import numpy as np

# Audio clip (16 kHz mono)
rate = 16000
audio_clip = np.zeros(2 * rate, dtype=np.float32)

questions_audio = {
    "has_speech": {"type": "boolean"},
    "speaker_count": {"type": "number", "range": [0, 5], "step": 1}
}

audio_decision = model.system_one(media={"audio": audio_clip}, questions=questions_audio)
print(audio_decision)

6. System 2: Generative Chat & Reasoning

input_tokens = torch.tensor([[101, 2054, 2003, 1037, 102]], device=device)
generated_tokens = model.generate(input_tokens, max_new_tokens=32)
print("Generated Tokens:", generated_tokens.tolist())

πŸ“Š Hardware & Resource Requirements

Format / Dtype VRAM / RAM Usage Recommended Target
BF16 / FP32 (Default) ~4.5 GB NVIDIA RTX 3060 / 4060 (8GB+), Mac M1-M4 (8GB+)
4-bit Quantized (GGUF/AWQ) ~1.8 GB Raspberry Pi 5, Edge NPU, Mobile Devices

πŸ“œ Citation & Credits

Special thanks to Liquid AI for training and open-sourcing the base foundation models:

  • LiquidAI/LFM3-Omni-2B
  • LiquidAI/d1-Omni-2B
@misc{lfm3_omni_dualhead_2026,
  author = {JonusNattapong},
  title = {LFM3-Omni Dual-Head Architecture: System 1 Decision and System 2 Generation on a Shared Backbone},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/JonusNattapong/lfm3-omni-dualhead}}
}

πŸ“„ License

This project is released under the Apache 2.0 License.

Downloads last month
668
Safetensors
Model size
75M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for JonusNattapong/lfm3-omni-dualhead

Finetuned
(22)
this model