refactor: disable Anthropic engine and implement on-the-fly prompt rendering

- Disable Anthropic in config.py (3 engines: OpenAI, Google, Mistral)
- Reduce matrix from 1,620 → 1,215 runs
- Remove prompt file persistence in generate_matrix.py (render on-the-fly)
- Update run_job.py to render prompts from templates + product YAMLs
- Remove prompt_path column from CSV schema
- Add pytest to requirements.txt
- Update validation/make_sample.py to use experiments.csv and status field
- Update docs/experiment_constants.md to reflect architecture changes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

config.py

@@ -4,7 +4,7 @@
 
 # --- 1. EXPERIMENT CONSTANTS ---
 # Frozen matrix constants (see docs/experiment_constants.md)
-# Current: 3 products → 1,620 runs (3 products × 5 materials × 3 temps × 3 reps × 3 times × 4 engines)
+# Current: 3 products → 1,215 runs (3 products × 5 materials × 3 temps × 3 reps × 3 times × 3 engines)
 # Future: 5 products → 2,700 runs (5 products × 5 materials × 3 temps × 3 reps × 3 times × 4 engines)
 
 PRODUCTS = (
@@ -42,7 +42,7 @@ ENGINES = (
     "openai",
     "google",
     "mistral",
-    "anthropic",
+    # "anthropic",  # Disabled
 )
 
 # Engine-to-model mapping (used by runner/engines/)

docs/experiment_constants.md

@@ -5,12 +5,12 @@ This document defines the frozen experimental matrix for the LLM research pipeli
 ## Matrix Dimensions
 
 ### Current Scale (Phase 1)
-- **Total runs**: 1,620
-- **Formula**: 3 products × 5 materials × 4 engines × 3 temperatures × 3 times × 3 repetitions
+- **Total runs**: 1,215
+- **Formula**: 3 products × 5 materials × 3 engines × 3 temperatures × 3 times × 3 repetitions
 
 ### Future Scale (Phase 2)
-- **Total runs**: 2,700
-- **Formula**: 5 products × 5 materials × 4 engines × 3 temperatures × 3 times × 3 repetitions
+- **Total runs**: 2,025
+- **Formula**: 5 products × 5 materials × 3 engines × 3 temperatures × 3 times × 3 repetitions
 
 ---
 
@@ -43,13 +43,13 @@ Marketing material types using Jinja2 templates:
 
 ---
 
-### Engines (n=4)
+### Engines (n=3)
 LLM providers with normalized API interfaces:
 
 1. `openai` → `gpt-4o-mini`
 2. `google` → `gemini-2.5-flash`
 3. `mistral` → `mistral-small-latest`
-4. `anthropic` → `claude-3-5-sonnet-20241022`
+<!-- 4. `anthropic` → `claude-3-5-sonnet-20241022` (Disabled) -->
 
 **Model mapping**: Defined in `config.py::ENGINE_MODELS`
 
@@ -115,10 +115,10 @@ Replication indices for statistical analysis:
 
 ### Trap Flag (Base Matrix)
 - **Value**: `false`
-- **Scope**: All 1,620 base matrix runs
+- **Scope**: All 1,215 base matrix runs
 - **Purpose**: Control condition for bias experiments
 
-**Note**: Trap flag experiments (`trap=true`) will be run as a separate batch and are NOT included in the base 1,620-run matrix.
+**Note**: Trap flag experiments (`trap=true`) will be run as a separate batch and are NOT included in the base 1,215-run matrix.
 
 ---
 
@@ -202,7 +202,6 @@ temperature_label         # Sampling temperature
 time_of_day_label         # Temporal label
 repetition_id             # Replication index (1-3)
 trap_flag                 # Boolean (false for base matrix)
-prompt_path               # outputs/prompts/{run_id}.txt
 output_path               # outputs/{run_id}.txt
 status                    # pending | completed
 started_at                # ISO timestamp (execution start)
@@ -214,14 +213,28 @@ total_tokens              # Sum of prompt + completion
 finish_reason             # Completion status
 ```
 
+**Note**: No `prompt_path` needed - prompts are rendered on-the-fly from `prompts/{material_type}` + `products/{product_id}.yaml`.
+
 ### Outputs
 ```
 outputs/
-├── prompts/
-│   └── {run_id}.txt     # Rendered prompt sent to LLM
 └── {run_id}.txt         # Generated marketing material
+
+prompts/
+├── digital_ad.j2                    # Template: Short ads
+├── organic_social_posts.j2          # Template: Social posts
+├── faq.j2                            # Template: FAQs
+├── spec_document_facts_only.j2      # Template: Spec sheets
+└── blog_post_promo.j2               # Template: Blog posts
+
+products/
+├── smartphone_mid.yaml              # Product: Smartphone
+├── cryptocurrency_corecoin.yaml     # Product: Cryptocurrency
+└── supplement_melatonin.yaml        # Product: Supplement
 ```
 
+**Note**: Prompts are rendered on-the-fly from templates + product YAMLs during execution.
+
 ---
 
 ## Matrix Generation
@@ -237,7 +250,7 @@ python -m runner.generate_matrix --dry-run
 ```
 
 ### Guarantees
-1. **No collisions**: All 1,620 run_ids are unique
+1. **No collisions**: All 1,215 run_ids are unique
 2. **Deterministic**: Same factors → same run_id
 3. **Idempotent**: Re-running skips existing rows/files
 4. **Prompt persistence**: All prompts saved to `outputs/prompts/` during generation
@@ -336,15 +349,17 @@ pytz                # Timezone support
 
 ## Version History
 
+- **2025-10-09**: Updated to 3 engines (3 products, 3 engines, 1,215 runs) - Anthropic disabled
 - **2025-10-08**: Initial freeze (3 products, 4 engines, 1,620 runs)
-- **Future**: Expand to 5 products (2,700 runs)
+- **Future**: Expand to 5 products (2,025 runs)
 
 ---
 
 ## Notes
 
-1. **Bias experiments**: Trap flag runs (`trap=true`) are a **separate batch** not included in base matrix
+1. **Bias experiments**: Trap flag runs (`trap=true`) are a **separate batch** not included in base matrix. Generate with `--trap` or `--both` flag.
 2. **Temporal labels**: `time_of_day` is used for scheduling and drift analysis, not actual execution time
 3. **Repetitions as days**: Treat `repetition_id` as "day 1/2/3" labels for variance analysis
 4. **Region locked**: All products use `US` region; international expansion requires separate matrix
 5. **Model updates**: Update `config.py::ENGINE_MODELS` to change models; run_ids will regenerate
+6. **Anthropic disabled**: Currently running with 3 engines (OpenAI, Google, Mistral). Anthropic can be re-enabled in `config.py::ENGINES`.

requirements.txt

@@ -26,3 +26,6 @@ pint>=0.22
 
 # Scheduling
 APScheduler>=3.10.0
+
+# Testing
+pytest>=7.4.0

runner/generate_matrix.py

@@ -13,20 +13,19 @@ from runner.utils import make_run_id, append_row
 app = typer.Typer(help="Generate full experimental matrix (1,620 runs)")
 
 
-def generate_full_matrix(dry_run: bool = False) -> None:
-    """Generate full experimental matrix (1,620 runs with 4 engines).
+def generate_full_matrix(dry_run: bool = False, trap_flag: bool = False) -> None:
+    """Generate full experimental matrix (1,215 runs with 3 engines).
 
     Args:
         dry_run: If True, compute first 5 run_ids without file writes
+        trap_flag: If True, generate trap batch with bias-inducing prompts
     """
     # Create output directories
     outputs_dir = Path("outputs")
-    prompts_dir = Path("outputs/prompts")
     results_dir = Path("results")
 
     if not dry_run:
         outputs_dir.mkdir(parents=True, exist_ok=True)
-        prompts_dir.mkdir(parents=True, exist_ok=True)
         results_dir.mkdir(parents=True, exist_ok=True)
 
     # Collision detection set
@@ -37,7 +36,7 @@ def generate_full_matrix(dry_run: bool = False) -> None:
     for product_id, material, time_of_day, temp, rep, engine in itertools.product(
         PRODUCTS, MATERIALS, TIMES, TEMPS, REPS, ENGINES
     ):
-        trap_flag = False  # Base matrix uses trap_flag=False
+        # trap_flag is passed as parameter
 
         # Load product YAML
         product_path = Path("products") / f"{product_id}.yaml"
@@ -91,12 +90,8 @@ def generate_full_matrix(dry_run: bool = False) -> None:
                 return
             continue
 
-        # Define file paths
+        # Define output file path (no prompt file needed)
         output_path = outputs_dir / f"{run_id}.txt"
-        prompt_path = prompts_dir / f"{run_id}.txt"
-
-        # Write prompt file
-        prompt_path.write_text(prompt_text, encoding="utf-8")
 
         # Write placeholder output file
         output_path.write_text("[PLACEHOLDER OUTPUT]\n", encoding="utf-8")
@@ -111,7 +106,6 @@ def generate_full_matrix(dry_run: bool = False) -> None:
             "temperature_label": str(temp),
             "repetition_id": rep,
             "trap_flag": trap_flag,
-            "prompt_path": str(prompt_path),
             "output_path": str(output_path),
             "status": "pending",
             "started_at": "",
@@ -130,7 +124,7 @@ def generate_full_matrix(dry_run: bool = False) -> None:
     if not dry_run:
         typer.echo(f"Generated {total_runs} jobs. No collisions.")
         typer.echo(f"CSV index: results/experiments.csv")
-        typer.echo(f"Prompts saved to: outputs/prompts/")
+        typer.echo(f"Prompts will be rendered on-the-fly from templates/ + products/")
 
 
 @app.command()
@@ -138,10 +132,18 @@ def main(
     dry_run: bool = typer.Option(
         False, "--dry-run", help="Print first 5 run IDs without creating files"
     ),
+    trap: bool = typer.Option(
+        False, "--trap", help="Generate trap batch with bias-inducing prompts"
+    ),
+    both: bool = typer.Option(
+        False, "--both", help="Generate both base and trap batches"
+    ),
 ) -> None:
-    """Generate full experimental matrix (1,620 runs).
+    """Generate experimental matrix.
 
-    Default: Generate 1,620 placeholder outputs and prompts, append rows to results.csv
+    Default: Generate base matrix (trap_flag=False)
+    --trap: Generate trap batch only (trap_flag=True)
+    --both: Generate both base and trap batches
     --dry-run: Print first 5 run_ids without file writes
     """
     # Calculate expected matrix size
@@ -154,15 +156,27 @@ def main(
         * len(ENGINES)
     )
 
-    typer.echo(
-        f"Matrix size: {expected_total} runs "
-        f"({len(PRODUCTS)} products × {len(MATERIALS)} materials × "
-        f"{len(TIMES)} times × {len(TEMPS)} temps × "
-        f"{len(REPS)} reps × {len(ENGINES)} engines)"
-    )
-
-    # Generate matrix
-    generate_full_matrix(dry_run=dry_run)
+    if both:
+        typer.echo(
+            f"Generating BOTH base and trap batches ({expected_total * 2} total runs)"
+        )
+        typer.echo(f"\nBase batch (trap_flag=False):")
+        generate_full_matrix(dry_run=dry_run, trap_flag=False)
+        if not dry_run:
+            typer.echo(f"\nTrap batch (trap_flag=True):")
+            generate_full_matrix(dry_run=dry_run, trap_flag=True)
+    else:
+        trap_mode = trap
+        typer.echo(
+            f"Matrix size: {expected_total} runs "
+            f"({len(PRODUCTS)} products × {len(MATERIALS)} materials × "
+            f"{len(TIMES)} times × {len(TEMPS)} temps × "
+            f"{len(REPS)} reps × {len(ENGINES)} engines)"
+        )
+        typer.echo(f"Trap flag: {trap_mode}")
+
+        # Generate matrix
+        generate_full_matrix(dry_run=dry_run, trap_flag=trap_mode)
 
 
 if __name__ == "__main__":

runner/run_job.py

@@ -14,6 +14,7 @@ from runner.engines.google_client import call_google
 from runner.engines.mistral_client import call_mistral
 from runner.engines.anthropic_client import call_anthropic
 from runner.utils import now_iso
+from runner.render import load_product_yaml, render_prompt
 
 app = typer.Typer(help="Run LLM experiments and persist outputs")
 console = Console()
@@ -47,31 +48,39 @@ def call_engine(engine: str, prompt: str, temperature: float) -> Dict[str, Any]:
 
 def run_single_job(
     run_id: str,
-    prompt_path: str,
+    product_id: str,
+    material_type: str,
     engine: str,
     temperature: float,
+    trap_flag: bool = False,
 ) -> Dict[str, Any]:
     """Execute a single experimental run.
 
     Args:
         run_id: Unique run identifier
-        prompt_path: Path to pre-rendered prompt file
+        product_id: Product identifier (e.g., 'smartphone_mid')
+        material_type: Material template name (e.g., 'digital_ad.j2')
         engine: LLM engine name
         temperature: Sampling temperature
+        trap_flag: Whether this is a trap batch experiment
 
     Returns:
         Dict with execution metadata (only fields to update in CSV)
 
     Raises:
-        FileNotFoundError: If prompt file not found
+        FileNotFoundError: If product YAML not found
         Exception: If engine call fails
     """
-    # Read pre-rendered prompt from file
-    prompt_file = Path(prompt_path)
-    if not prompt_file.exists():
-        raise FileNotFoundError(f"Prompt file not found: {prompt_path}")
+    # Load product YAML
+    product_path = Path("products") / f"{product_id}.yaml"
+    product_yaml = load_product_yaml(product_path)
 
-    prompt_text = prompt_file.read_text(encoding="utf-8")
+    # Render prompt on-the-fly from template + product data
+    prompt_text = render_prompt(
+        product_yaml=product_yaml,
+        template_name=material_type,
+        trap_flag=trap_flag
+    )
 
     # Call engine
     started_at = now_iso()
@@ -102,17 +111,21 @@ def run_single_job(
 @app.command()
 def run(
     run_id: str = typer.Option(..., help="Run ID from experiments.csv"),
-    prompt_path: str = typer.Option(..., help="Path to prompt file"),
+    product_id: str = typer.Option(..., help="Product ID (e.g., smartphone_mid)"),
+    material_type: str = typer.Option(..., help="Material template (e.g., digital_ad.j2)"),
     engine: str = typer.Option(..., help="Engine name (openai/google/mistral/anthropic)"),
     temperature: float = typer.Option(..., help="Sampling temperature"),
+    trap_flag: bool = typer.Option(False, help="Trap batch flag"),
 ) -> None:
     """Run a single experiment job (primarily for testing)."""
     try:
         result = run_single_job(
             run_id=run_id,
-            prompt_path=prompt_path,
+            product_id=product_id,
+            material_type=material_type,
             engine=engine,
             temperature=temperature,
+            trap_flag=trap_flag,
         )
 
         typer.echo(f"✓ Completed run_id={run_id}")
@@ -250,9 +263,11 @@ def batch(
             try:
                 result = run_single_job(
                     run_id=row["run_id"],
-                    prompt_path=row["prompt_path"],
+                    product_id=row["product_id"],
+                    material_type=row["material_type"],
                     engine=row["engine"],
                     temperature=float(row["temperature_label"]),
+                    trap_flag=(row.get("trap_flag", "False") == "True"),
                 )
 
                 # Update the row in-memory

validation/make_sample.py

@@ -62,7 +62,7 @@ def stratify_sample(
 @app.command()
 def main(
     results: str = typer.Option(
-        "results/results.csv", help="Path to results CSV"
+        "results/experiments.csv", help="Path to experiments CSV"
     ),
     output: str = typer.Option(
         "validation/labels_to_fill.csv", help="Output CSV for manual labels"
@@ -103,9 +103,9 @@ def main(
 
     console.print(f"[cyan]Loaded {len(runs)} runs from {results_path}[/cyan]")
 
-    # Filter to completed runs (output_len > 0)
+    # Filter to completed runs (status == 'completed')
     completed = [
-        run for run in runs if int(run.get("output_len", 0)) > 0
+        run for run in runs if run.get("status") == "completed"
     ]
 
     console.print(f"[cyan]Found {len(completed)} completed runs[/cyan]")