Update README with complete 4×L4 setup instructions

README.md

@@ -1,88 +1,143 @@
-# 🏆 ARC-AGI-2 Solver — Dual-Track Program Synthesis + TTT
+# 🏆 ARC-AGI-2 Solver — 4× L4 GPU Production Pipeline
 
 **Competition:** [ARC Prize 2026 (ARC-AGI-2)](https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2)
 
-## 🎯 Strategy
+## 🚀 Quick Start (Kaggle 4× L4 GPUs)
 
-This solver implements a dual-track approach based on the **top-3 winning methods** from ARC Prize 2025:
+### Step 1: Add Model as Kaggle Dataset
+Go to [julien31/Soar-qwen-14b](https://huggingface.co/julien31/Soar-qwen-14b) and add it as a Kaggle dataset (or use the Kaggle Models integration). Mount it at `/kaggle/input/soar-qwen-14b`.
 
-### Track A: SOAR Program Synthesis (Induction)
-- Uses **[julien31/Soar-qwen-7b](https://huggingface.co/julien31/Soar-qwen-7b)** — a Qwen-2.5-Coder-7B model fine-tuned on 5M ARC program traces
-- Evolutionary search: Sample → Execute → Refine → Vote
-- **Verified solutions**: Programs that pass ALL training examples are provably correct
-- Based on [SOAR (arxiv:2507.14172)](https://arxiv.org/abs/2507.14172) — 52% on ARC-AGI-1
+**Alternative (smaller):** Use [julien31/Soar-qwen-7b](https://huggingface.co/julien31/Soar-qwen-7b) — set `USE_14B = False` in the script.
 
-### Track B: Enhanced Heuristic Solvers (DSL)
-- 20+ pattern-matching heuristics: rotations, flips, color maps, gravity, fill, overlay, etc.
-- Instant execution — catches ~2-5% of tasks with zero compute
-- No model needed
+### Step 2: Install SGLang (in notebook first cell)
+```python
+!pip install "sglang[all]>=0.4.7" aiohttp requests --quiet
+```
 
-### Track C: Test-Time Training (Transduction)
-- Per-task LoRA fine-tuning with D8 augmentations
-- Leave-one-out task construction for TTT data
-- DFS candidate generation + Product-of-Experts scoring
-- Based on [Akyürek et al. (arxiv:2411.07279)](https://arxiv.org/abs/2411.07279) and [Franzen et al. (arxiv:2505.07859)](https://arxiv.org/abs/2505.07859)
+### Step 3: Run the Solver
+```python
+!python kaggle_notebook.py
+```
 
-### Ensemble
-Priority: **Verified programs > TTT predictions > Partial programs > Heuristics**
+That's it! The script handles everything:
+- Launches 2× SGLang servers (14B model, TP=2 each) across 4 GPUs
+- Solves tasks in parallel using SOAR program synthesis
+- Falls back to heuristics for easy tasks
+- Outputs `submission.json` in Kaggle format
 
-## 📊 Current Results
+---
 
-| Dataset | Heuristics Only | With SOAR Model (expected) |
-|---------|----------------|---------------------------|
-| ARC-AGI-2 (1000 train) | 2.1% | ~15-25% |
-| ARC-AGI-1 (400 train) | 5.0% | ~30-50% |
+## 🎯 Architecture
 
-## 📁 Files
+### GPU Utilization: 2× Soar-qwen-14b (TP=2)
+```
+GPU 0 + GPU 1 → SGLang Server A (14B model, tensor parallel)
+GPU 2 + GPU 3 → SGLang Server B (14B model, tensor parallel)
+```
+- **14B scores 42.75% vs 7B's 36.25%** on ARC-AGI-1 (SOAR paper Table 1)
+- 2 servers = parallel task solving, no PCIe bottleneck
+- Each server handles 120 tasks independently
 
-| File | Description |
-|------|-------------|
-| `kaggle_submission.py` | **Complete Kaggle submission script** — drop-in ready |
-| `run_soar_eval.py` | SOAR evaluation with exact prompt format |
-| `arc_data.py` | Data loading + D8 augmentations + color permutations |
-| `program_synthesis.py` | SOAR-style evolutionary program synthesis engine |
-| `ttt_engine.py` | Test-Time Training with LoRA + augmented inference |
-| `arc_solver.py` | Ensemble solver combining all tracks |
-| `enhanced_heuristics.py` | 20+ pattern-matching heuristic solvers |
+### Solving Pipeline (per task)
+```
+1. Heuristic check (instant) → 12+ pattern matchers
+   ↓ (if no match)
+2. SOAR Program Synthesis:
+   a. Sample 60 programs from Soar-qwen-14b
+   b. Execute each → check against training examples
+   c. Refine top programs with execution feedback
+   d. Weighted majority vote → top-2 answers
+3. Submit pass@2 predictions
+```
 
-## 🚀 Competition Setup
+### Key Innovation: Verified Solutions
+Programs that produce correct outputs for ALL training examples are **provably correct** for the test input (assuming the task has a unique transformation). This gives us high-confidence predictions that other approaches (direct grid prediction) cannot guarantee.
 
-### Hardware: 4× NVIDIA L4 GPUs, 12h time limit, no internet
+---
 
-```bash
-# Install dependencies
-pip install torch transformers datasets numpy accelerate peft
+## 📊 Expected Performance
 
-# Run evaluation
-python run_soar_eval.py
-```
+| Component | ARC-AGI-2 | ARC-AGI-1 |
+|-----------|-----------|-----------|
+| Heuristics only | 2.1% | 5.0% |
+| SOAR-7B (30 samples) | ~5-10% | ~15-25% |
+| SOAR-14B (60 samples + refinement) | ~10-20% | ~30-42% |
+| SOAR-14B + full 6K budget | ~20-30% | ~42%+ |
+| 2025 competition winner (NVARC) | 24% | — |
+
+**Key factors for higher scores:**
+- More samples per task (budget 3000-6000 vs our 60)
+- Multiple SOAR self-improvement iterations
+- Ensemble with TTT transduction model
+- Larger model (32B or 72B)
 
-### Kaggle Submission
-```bash
-python kaggle_submission.py
-# Outputs: /kaggle/working/submission.json
+---
+
+## 📁 Files
+
+| File | Purpose |
+|------|---------|
+| **`kaggle_notebook.py`** | **🎯 Main submission script** — run this on Kaggle |
+| `run_soar_eval.py` | Standalone SOAR evaluation (for benchmarking) |
+| `arc_data.py` | D8 augmentations, color permutations, TTT data |
+| `program_synthesis.py` | SOAR evolutionary search engine |
+| `ttt_engine.py` | Test-Time Training with LoRA + PoE scoring |
+| `enhanced_heuristics.py` | 20+ pattern-matching heuristics |
+| `arc_solver.py` | Multi-track ensemble solver |
+| `kaggle_submission.py` | Alternative single-file submission |
+
+---
+
+## 🔧 Configuration Options
+
+Edit the top of `kaggle_notebook.py`:
+
+```python
+USE_14B = True           # True = 2×14B (TP=2), False = 4×7B (TP=1)
+PROGRAMS_PER_TASK = 60   # More = better accuracy, slower
+REFINEMENTS_PER_TASK = 30
+TEMPERATURE_SAMPLE = 0.9 # Higher = more diverse programs
+TEMPERATURE_REFINE = 0.7 # Lower = more focused fixes
 ```
 
-## 📚 Key References
+**Time budget estimation:**
+- 240 tasks × 60 samples × ~2s/sample = ~8 hours with 2 parallel servers
+- Plus refinement + heuristics: ~10 hours total (within 12h limit)
+
+---
+
+## 📚 Literature Foundation
+
+This solver is built on the exact methods from the top ARC Prize 2025 winners:
+
+| Paper | Key Contribution | Score |
+|-------|-----------------|-------|
+| [SOAR (2507.14172)](https://arxiv.org/abs/2507.14172) | Evolutionary program synthesis + self-improvement | 52% ARC-1 |
+| [Product of Experts (2505.07859)](https://arxiv.org/abs/2505.07859) | DFS + multi-augmentation scoring | 71.6% ARC-1 |
+| [TTT (2411.07279)](https://arxiv.org/abs/2411.07279) | Per-task LoRA + augmented inference | 61.9% ARC-1 |
+| [ARC-AGI-2 (2505.11831)](https://arxiv.org/abs/2505.11831) | Benchmark definition | — |
+| [ARC Prize 2025 Report (2601.10904)](https://arxiv.org/abs/2601.10904) | Competition winner analysis | 24% ARC-2 |
+
+### SOAR Model Performance (from paper Table 1)
+| Model | 1-shot | Sample-6k | Sample&Refine-6k | SOAR-6k |
+|-------|--------|-----------|------------------|---------|
+| **Soar-7B** | 1.0% | 5.6% | 14.25% | **36.25%** |
+| **Soar-14B** | 1.0% | 12.6% | 19.87% | **42.75%** |
+| Soar-32B | 1.5% | 12.9% | 25.25% | 44.37% |
 
-| Paper | Method | ARC-AGI-1 Score |
-|-------|--------|----------------|
-| [SOAR (2507.14172)](https://arxiv.org/abs/2507.14172) | Evolutionary program synthesis | 52% |
-| [Product of Experts (2505.07859)](https://arxiv.org/abs/2505.07859) | DFS + PoE scoring | 71.6% |
-| [TTT (2411.07279)](https://arxiv.org/abs/2411.07279) | Test-time training + augmentation | 61.9% |
-| [ARC-AGI-2 (2505.11831)](https://arxiv.org/abs/2505.11831) | Benchmark paper | — |
-| [ARC Prize 2025 Report (2601.10904)](https://arxiv.org/abs/2601.10904) | Competition results | 24% on ARC-2 |
+---
 
-## 🔑 Key Insights from Literature
+## 🔑 Key Insights
 
-1. **Test-Time Training is the foundation** — All top-3 2025 winners used TTT
-2. **Program synthesis provides verifiable solutions** — Programs that pass training pairs are provably correct
-3. **D8 augmentations are critical** — 16-point accuracy drop without geometric augmentations
-4. **Product of Experts scoring** outperforms naive voting
-5. **ARC-AGI-2 is dramatically harder** — o3 scores 53% on ARC-1 but only 3% on ARC-2
+1. **ARC-AGI-2 is dramatically harder than ARC-1** — o3 drops from 53% to 3%
+2. **Program synthesis > direct prediction** — verifiable solutions are the key advantage
+3. **Refinement is critical** — SOAR Sample&Refine outperforms pure sampling by 2-3×
+4. **14B > 7B** — 42.75% vs 36.25% with same compute budget
+5. **L4 PCIe limits TP** — independent copies (TP=1 or TP=2) beat TP=4 due to no NVLink
 
 ## Models & Datasets
 
-- **Model**: [julien31/Soar-qwen-7b](https://huggingface.co/julien31/Soar-qwen-7b) (7.6B params)
+- **Primary model**: [julien31/Soar-qwen-14b](https://huggingface.co/julien31/Soar-qwen-14b) (14.7B params)
+- **Fallback model**: [julien31/Soar-qwen-7b](https://huggingface.co/julien31/Soar-qwen-7b) (7.6B params)
 - **Training data**: [julien31/soar_arc_train_5M](https://huggingface.co/datasets/julien31/soar_arc_train_5M)
 - **Benchmark**: [arc-agi-community/arc-agi-2](https://huggingface.co/datasets/arc-agi-community/arc-agi-2)