Initial model card and config for semsimula-parflm-multixi

README.md

@@ -0,0 +1,259 @@
+---
+pipeline_tag: text-generation
+library_name: pytorch
+language:
+  - en
+license: cc-by-4.0
+tags:
+  - research
+  - semsimula
+  - conservative-language-model
+  - scalar-potential
+  - lagrangian-mechanics
+  - energy-based-model
+  - physics-informed
+  - parflm
+  - parf
+  - multi-channel-xi
+  - sparse-routing
+  - tinystories
+datasets:
+  - roneneldan/TinyStories
+metrics:
+  - perplexity
+model-index:
+  - name: semsimula-parflm-multixi
+    results:
+      - task:
+          type: text-generation
+        dataset:
+          type: roneneldan/TinyStories
+          name: TinyStories
+          split: validation
+        metrics:
+          - type: perplexity
+            value: 12.06
+            name: Validation Perplexity
+---
+
+# Multi-Xi PARFLM (Property-Attractive-Repulsive Force Language Model)
+
+The **Multi-Xi PARFLM** extends the SPLM with **pairwise token interaction forces** derived from a second scalar potential V_phi. While the base SPLM's V_theta provides a single-body potential (each token interacts only with a summary of its past), PARFLM adds explicit pairwise forces V_phi(h_t, h_s) between tokens -- the physics-informed analogue of attention's pairwise dot-product, but derived from a gradient of a scalar potential (making it conservative).
+
+The pairwise forces use **Gumbel-softmax top-k sparse routing** to keep the cost at O(T*k) rather than O(T^2). This model achieves **12.06 PPL** on TinyStories, a 2.6 PPL improvement over the standalone Multi-Xi SPLM.
+
+Part of the [Semantic Simulation](https://doi.org/10.5281/zenodo.20579593) framework.
+
+## Table of Contents
+
+- [Model Details](#model-details)
+- [Architecture](#architecture)
+- [How to Get Started](#how-to-get-started)
+- [Training Details](#training-details)
+- [Evaluation Results](#evaluation-results)
+- [SPLM Family Overview](#splm-family-overview)
+- [Bias, Risks, and Limitations](#bias-risks-and-limitations)
+- [Citation](#citation)
+- [Environmental Impact](#environmental-impact)
+
+## Model Details
+
+### Model Description
+
+The Multi-Xi PARFLM combines two SPLM extensions:
+
+1. **Multi-channel K-EMA xi** (from the Multi-Xi SPLM): K=8 learnable causal exponential moving averages giving V_theta a multi-resolution summary of the past.
+2. **Sparse PARF pair-interactions:** A second scalar potential V_phi(h_t, h_s) adds particle-exchange forces between token pairs, routed via Gumbel-softmax top-k selection.
+
+The total potential energy for token t is:
+
+    U_t = V_theta(xi_t, h_t) + sum_{s<t} m_ts * V_phi(h_t, h_s)
+
+and the conservative force is f_t = -grad_{h_t} U_t.
+
+- **Developed by:** Dimitar P. Gueorguiev (Independent Researcher)
+- **Model type:** Conservative autoregressive language model with pairwise forces
+- **Language:** English
+- **License:** [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)
+
+### Model Sources
+
+- **Paper:** [Semantic Simulation: A Prescriptive Lagrangian Framework for Efficient Semantic Inference](https://doi.org/10.5281/zenodo.20579593)
+- **Repository:** [github.com/dimitarpg13/semsimula-paper](https://github.com/dimitarpg13/semsimula-paper)
+- **Model source code:** [`notebooks/conservative_arch/parf/model_parf_multixi.py`](https://github.com/dimitarpg13/semsimula-paper/blob/main/notebooks/conservative_arch/parf/model_parf_multixi.py)
+
+## Architecture
+
+```
+Input tokens x_1, ..., x_T
+       |
+   Embedding E[x] + positional encoding
+       |
+   For each of L=8 integration steps:
+       |
+       +-- K-EMA channels: xi^(k)_t = causal_ema(h, alpha_k)   [K=8 channels]
+       |
+       +-- Single-body: V_theta([xi_1..xi_K, h]) -> R           [3-layer MLP]
+       |
+       +-- Pair routing: score_head(h_t, h_s) -> top-k selection [Gumbel-softmax]
+       |
+       +-- Pair forces: V_phi(h_t, h_s) -> R                    [structural competitive]
+       |
+       +-- Total: U_t = V_theta + sum V_phi
+       |
+       +-- Conservative force: f = -grad_h U_t                  [autograd]
+       |
+       +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
+       |
+       +-- LayerNorm(h)
+       |
+   Logits = h @ E^T                                            [tied embeddings]
+```
+
+| Parameter | Value |
+|---|---|
+| Hidden dim (d) | 256 |
+| Layers (L) | 8 |
+| V_theta hidden / depth | 1024 / 3 |
+| Xi channels (K) | 8 |
+| Alpha init | log-spaced |
+| V_phi kind | structural_competitive |
+| V_phi hidden (H) | 128 |
+| Sparse routing top_k | 8 |
+| Gumbel tau | 1.0 -> 0.1 (annealed) |
+| Mass model | logfreq (frozen surprisal lookup) |
+| Damping (gamma) | 0.30 (fixed) |
+| Total parameters | **17,632,215** |
+
+### Key Design Properties
+
+- **Globally conservative:** Both V_theta and V_phi are scalar potentials; the total force f = -grad(V_theta + sum V_phi) is conservative by construction.
+- **Sparse routing:** Gumbel-softmax top-k selection keeps pairwise cost at O(T*k) instead of O(T^2).
+- **Stage-1.5b gathered V_phi:** Memory-efficient implementation replacing O(T^2) intermediates with O(T*k).
+- **Inheritance chain:** MultiXiPARFLM -> SparsePARFLM -> PARFLM (all conservative).
+
+## How to Get Started
+
+```python
+# Clone the companion repository for full source code
+# git clone https://github.com/dimitarpg13/semsimula-paper.git
+# cd semsimula-paper/notebooks/conservative_arch
+
+import torch
+import sys
+sys.path.insert(0, "parf")
+sys.path.insert(0, "multixi")
+sys.path.insert(0, "energetic_minima")
+sys.path.insert(0, "sarf_mass_variant")
+
+from parf.model_parf_multixi import MultiXiPARFLM, MultiXiPARFConfig
+
+config = MultiXiPARFConfig(
+    vocab_size=50257,
+    d=256,
+    n_layers=8,
+    v_hidden=1024,
+    v_depth=3,
+    max_len=1024,
+    block_size=512,
+    gamma=0.30,
+    xi_channels=8,
+    xi_alpha_inits="log_spaced",
+    v_phi_kind="structural_competitive",
+    v_phi_hidden=128,
+    top_k=8,
+)
+
+model = MultiXiPARFLM(config)
+print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")
+
+# Forward pass
+x = torch.randint(0, 50257, (1, 64))
+logits, loss = model(x, targets=x)
+```
+
+## Training Details
+
+### Training Data
+
+[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) -- GPT-2 BPE tokenization. Training cap: 5M tokens.
+
+### Training Procedure
+
+| Hyperparameter | Value |
+|---|---|
+| Optimizer | AdamW |
+| Learning rate | 5e-4 (cosine decay) |
+| Warmup steps | 400 |
+| Weight decay | 0.01 |
+| Gradient clipping | 1.0 |
+| Batch size | 16 |
+| Block size | 512 |
+| Training steps | 8,000 |
+| Memory optimisation | Level-2 grad checkpoint + Stage-1.5b gathered V_phi |
+| Hardware | A100 40GB (Google Colab) |
+
+### Training Script
+
+[`notebooks/conservative_arch/scaleup/train_parf_multixi_scaleup.py`](https://github.com/dimitarpg13/semsimula-paper/blob/main/notebooks/conservative_arch/scaleup/train_parf_multixi_scaleup.py)
+
+## Evaluation Results
+
+### TinyStories Validation Perplexity
+
+| Model | PPL | Params | Gap vs Attention |
+|---|---|---|---|
+| Matched Attention (baseline) | **7.81** | 19.5M | -- |
+| [Hybrid SPLM+Attn](https://huggingface.co/dimitarpg13/semsimula-hybrid-splm) | **8.50** | ~19.0M | +0.69 |
+| [Fock-PARFLM v2.1](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) | **9.30** | 17.4M | +1.49 |
+| [Fock Attention](https://huggingface.co/dimitarpg13/semsimula-fock-attention) | **9.42** | 16.7M | +1.61 |
+| **Multi-Xi PARFLM** (this model) | **12.06** | 17.6M | +4.25 |
+| [Multi-Xi SPLM](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) | **14.69** | 16.5M | +6.88 |
+
+Adding sparse pairwise forces (V_phi) improves PPL from 14.69 to 12.06 over the standalone Multi-Xi SPLM, confirming that pairwise token interactions are a necessary complement to the single-body potential. The remaining gap to attention is closed further by the Fock register mechanism.
+
+## SPLM Family Overview
+
+This model is part of the **Semantic Simulation SPLM family**:
+
+| Model | Design | HuggingFace |
+|---|---|---|
+| Multi-Xi SPLM | Pure scalar potential, K-EMA context | [semsimula-splm-multixi](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) |
+| Hybrid SPLM+Attn | Attention front-end + SPLM refinement | [semsimula-hybrid-splm](https://huggingface.co/dimitarpg13/semsimula-hybrid-splm) |
+| Multi-Xi PARFLM | Scalar potential + sparse pairwise forces | [this model](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi) |
+| Fock-PARFLM v2.1 | PARFLM + Fock register pool (mediated exchange) | [semsimula-fock-parflm](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) |
+| Fock Attention | PARFLM + direct token-to-token exchange | [semsimula-fock-attention](https://huggingface.co/dimitarpg13/semsimula-fock-attention) |
+
+## Bias, Risks, and Limitations
+
+- **Research checkpoint only.** Proof-of-concept for the conservative pairwise-force architecture.
+- **TinyStories only.** Trained exclusively on synthetic children's stories (~5M tokens).
+- **English only.** No multilingual capability.
+- **Small scale.** 17.6M parameters, 256-dim hidden states.
+- **No safety training.** No RLHF, DPO, or safety filtering has been applied.
+
+## Citation
+
+```bibtex
+@misc{Gueorguiev2026SemSim,
+  author    = {Gueorguiev, Dimitar P.},
+  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
+               for Efficient Semantic Inference --- A Conservative-by-
+               Construction Language Model and the Shared-Potential
+               Separator, with a Correspondence to Joint Embedding
+               Predictive Architectures},
+  year      = {2026},
+  publisher = {Zenodo},
+  doi       = {10.5281/zenodo.20579593},
+  url       = {https://doi.org/10.5281/zenodo.20579593},
+  note      = {Version v15 (Jun 7, 2026).
+               Companion code repository (DOI 10.5281/zenodo.20579561):
+               \url{https://github.com/dimitarpg13/semsimula-paper}}
+}
+```
+
+## Environmental Impact
+
+- **Hardware:** NVIDIA A100 40GB (Google Colab)
+- **Training time:** ~6 hours (8,000 steps with gradient checkpointing)
+- **Carbon footprint:** Estimated < 2 kg CO2

config.json

@@ -0,0 +1,28 @@
+{
+  "model_type": "MultiXiPARFLM",
+  "model_family": "semsimula-splm",
+  "vocab_size": 50257,
+  "d": 256,
+  "n_layers": 8,
+  "v_hidden": 1024,
+  "v_depth": 3,
+  "max_len": 1024,
+  "block_size": 512,
+  "gamma": 0.30,
+  "xi_channels": 8,
+  "xi_alpha_inits": "log_spaced",
+  "v_phi_kind": "structural_competitive",
+  "v_phi_hidden": 128,
+  "top_k": 8,
+  "gumbel_tau_start": 1.0,
+  "gumbel_tau_end": 0.1,
+  "mass_model": "logfreq",
+  "integrator": "semi_implicit_euler",
+  "ln_after_step": true,
+  "use_gathered_v_phi": true,
+  "use_layer_checkpoint": true,
+  "total_parameters": 17632215,
+  "best_val_ppl": 12.06,
+  "training_steps": 8000,
+  "dataset": "roneneldan/TinyStories"
+}