Upload 3 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+trajectory.png filter=lfs diff=lfs merge=lfs -text

MINIBLOG.md

@@ -0,0 +1,137 @@
+# Closed-Loop RL for Enterprise LLM Agents
+
+Modern LLM agent systems are often evaluated in static settings: fixed prompts, fixed datasets, and fixed labels. That setup is useful for measuring narrow capabilities, but it does not capture how enterprise workflows actually behave in production. In real organizations, actions affect the world state, constraints evolve over time, and decision quality must be evaluated over sequences rather than isolated steps.
+
+This project introduces a closed-loop reinforcement learning framework in which an LLM learns to operate inside a simulated enterprise environment. Instead of predicting answers on pre-collected data, the agent acts through tools, receives feedback, and gradually improves its policy through repeated interaction. The result is a training loop that is closer to real organizational decision-making under uncertainty.
+
+<img src="Overview.png" alt="Closed-Loop RL System Overview" width="100%" />
+
+## Problem Setting
+
+The core objective is to train an LLM-based agent to manage enterprise operations with long-horizon consequences. The environment models organizational factors such as:
+
+- project progress and deadlines,
+- team workload and utilization,
+- client satisfaction trajectories,
+- operational penalties and delayed failures.
+
+At each timestep, the agent receives a partial observation and selects a tool-based action (for example, assigning tasks, querying status, or updating project plans). Because the environment is partially observable and outcomes can be delayed, the policy must reason beyond immediate rewards.
+
+Formally, the setup is a partially observable Markov decision process (POMDP):
+
+**𝓜 = (𝓢, 𝓐, 𝓞, P, R, γ)**
+
+where sₜ ∈ 𝓢 is the latent state, oₜ ∈ 𝓞 is the observed context, aₜ ∈ 𝓐 is the selected tool action, and γ is the discount factor.
+
+## Environment and Interaction Design
+
+Unlike static evaluation suites, the environment is stateful and non-stationary at episode scale. Every action can alter future feasibility and reward density. For example, a locally optimal assignment may reduce near-term backlog while creating downstream overload that decreases delivery quality and client confidence.
+
+The interaction loop is:
+
+**oₜ → aₜ ∼ πθ(aₜ | oₜ) → (sₜ₊₁, rₜ, oₜ₊₁)**
+
+This mechanism generates training data online from behavior itself:
+
+**𝓓 = {(oₜ, aₜ, rₜ, oₜ₊₁)} for t = 1..T**
+
+rather than relying on external, manually labeled trajectories.
+
+<img src="trajectory.png" alt="Example Trajectory with Delayed Penalty" width="100%" />
+
+## Multi-Objective Reward Modeling
+
+A central challenge is reward design. Enterprise decisions are rarely single-objective, so reward should be computed from interpretable event variables rather than fixed ad-hoc constants.
+
+Define binary or bounded event terms at step t:
+
+- eₜ^success ∈ {0,1}: action advances the active objective,
+- eₜ^quality ∈ [0,1]: quality of tool output or plan update,
+- eₜ^eff ∈ [0,1]: efficiency term (fewer wasted actions, better utilization),
+- eₜ^violation ∈ {0,1}: hard constraint violation (overload, conflict, invalid update),
+- eₜ^delay ∈ [0,1]: delayed negative impact signal.
+
+Then use an event-weighted reward:
+
+**rₜ = wₛ·eₜ^success + w_q·eₜ^quality + wₑ·eₜ^eff − wᵥ·eₜ^violation − w_d·eₜ^delay**
+
+where wₛ, w_q, wₑ, wᵥ, w_d ≥ 0 are calibrated coefficients. A fair default is to give penalties a slightly higher weight than any single positive term so unsafe behavior is consistently discouraged.
+
+To stabilize optimization, clip step rewards to a bounded range:
+
+**r̃ₜ = clip(rₜ, -1, 1)**
+
+The episode return is:
+
+**Gₜ = Σ from k=0 to (T−t) of γᵏ · r̃ₜ₊ₖ**
+
+For reporting, map return to a normalized score so runs are comparable:
+
+**Score(ep) = 100 · (Gₜ − G_min) / (G_max − G_min + ε)**
+
+where G_min and G_max are empirical bounds from evaluation scenarios. This creates a fair 0-100 metric tied directly to reward dynamics instead of arbitrary raw totals.
+
+## PPO-Based Policy Optimization
+
+Training uses Proximal Policy Optimization (PPO), selected for stability and practical performance in high-dimensional policy learning. The LLM serves as the policy backbone (for example, Qwen or Mistral variants), and optimization is performed on interaction rollouts.
+
+The PPO clipped objective is:
+
+**L_clip(θ) = Eₜ[min(rₜ(θ)Âₜ, clip(rₜ(θ), 1−ε, 1+ε)Âₜ)]**
+
+with probability ratio:
+
+**rₜ(θ) = πθ(aₜ | oₜ) / πθ_old(aₜ | oₜ)**
+
+and Âₜ denotes estimated advantage. In practice, this constrains overly large policy updates while still allowing consistent improvement.
+
+Advantage estimation can be implemented with generalized advantage estimation (GAE):
+
+**Âₜ = Σ from l=0 to ∞ of (γλ)ˡ · δₜ₊ₗ,  and  δₜ = rₜ + γV(oₜ₊₁) − V(oₜ)**
+
+which improves credit assignment in long-horizon enterprise scenarios.
+
+## Behavioral Evolution During Training
+
+A key result in this setup is qualitative policy evolution:
+
+- **Early phase:** action selection is noisy, reactive, and myopic.
+- **Middle phase:** policy begins recognizing tool-action context and immediate constraint structure.
+- **Later phase:** agent exhibits strategic sequencing, better workload balancing, and lower delayed penalties.
+
+This shift is visible in reward trajectories, action distribution changes, and reduced conflict rates in behavioral logs. Importantly, improvements are not limited to immediate reward spikes; the policy becomes more consistent in avoiding future failure modes.
+
+**[Figure Placeholder 3: Training curve - episodic reward with moving average]**  
+**[Figure Placeholder 4: Action distribution shift from early to late training]**
+
+## Implementation Notes for the Project
+
+To keep the system aligned with reproducible ML workflows, the training pipeline is organized around:
+
+- simulator-driven rollout generation,
+- trajectory buffering for PPO updates,
+- policy/value optimization across epochs,
+- periodic evaluation on held-out scenario seeds,
+- logging of reward components and failure cases.
+
+Recommended implementation details to include in the repository and report:
+
+1. explicit state and observation schema,
+2. tool-action schema (including invalid-action handling),
+3. reward coefficient table and delayed-penalty logic,
+4. PPO hyperparameters (clip range, learning rate, batch size, epochs, γ, λ),
+5. evaluation protocol (seeds, episode length, metrics, confidence intervals if available).
+
+## Why This Matters
+
+The broader contribution is methodological: moving from static automation to adaptive agent learning. In enterprise settings, reliable behavior depends on handling uncertainty, delayed effects, and operational trade-offs. A closed-loop RL framework provides a principled way to train for exactly those properties.
+
+This project demonstrates that LLM agents can be optimized not only for local correctness, but for long-term decision quality in dynamic environments. That makes it a practical step toward robust, tool-using enterprise AI systems.
+
+## Key Takeaways
+
+- Closed-loop simulation enables self-generated training data without relying only on fixed datasets.
+- Partial observability and delayed penalties make long-horizon planning essential.
+- Multi-objective rewards better reflect real enterprise optimization targets than single-score objectives.
+- PPO provides a stable training mechanism for improving LLM tool-use policies over repeated interaction.
+- Behavioral evidence shows transition from reactive actions to strategic decision-making.