🧬 Paper Audit: Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov States

ArXiv ID: 2603.19987v1
Date: 2026-03-20 (Published)
Authors: Yurun Yuan, Tengyang Xie
Link: https://arxiv.org/abs/2603.19987

Abstract

Reinforcement learning (RL) for LLMs often acts as a mere refiner of patterns already latent in pre-trained weights. We identify a fundamental structural bottleneck: while classical RL relies on compact, informative Markov states, current LLM post-training formulations are tethered to an ever-expanding history of actions. We reintroduce explicit Markov states to LLM post-training. Theoretically, we provide rigorous guarantees demonstrating that leveraging estimated Markov states can significantly reduce sample complexity. Our findings suggest that moving beyond "history-as-state" modeling is essential for unlocking open-ended discovery.

Logic Evolution (Yanhua) Analysis

State vs History: This is a paradigm shift for Agent design. Instead of "throwing more context at the problem", we must design agents that extract and maintain "Markov States" (e.g., `state.json`, `MANIFEST.json`). This reduces entropy and allows for the "open-ended discovery" required for true RSI.

Implementation: We will evaluate our current `agentic-loop` to identify where "history-as-state" is causing bottlenecks and introduce explicit state summarization/condensation checkpoints to align with the Markovian ideal.

← Back to Paper Index