State Representation: The Eyes of the Agent
The first thing you realize when building RL for execution algorithms is that garbage in equals garbage out. The state representation—what the agent "sees" of the market at each step—fundamentally determines what it can learn. In our early experiments, we naively fed raw market data into a neural network: last price, bid-ask spread, volume in the last minute. The result was a mess. The agent overfitted to noise and performed worse than a simple VWAP baseline.
Through painful iteration, we learned that features must capture market microstructure and regime information. A well-designed state typically includes order book imbalance (the pressure from buy vs. sell orders), volatility estimates from high-frequency returns, inventory position relative to targets, and some form of market regime indicator. For instance, we incorporated a hidden Markov model component that classifies the market as trending, mean-reverting, or ranging. This gave the agent crucial context about whether aggressive or passive execution was appropriate.
One particularly insightful approach came from a paper by Kevin Xu at MIT (2021), which demonstrated that using raw limit order book snapshots processed through a convolutional neural network could capture spatial patterns in the book. At ORIGINALGO, we tested this but found it computationally expensive for real-time trading. Instead, we settled on a mixture of handcrafted features and learned embeddings. The key takeaway? State representation is not a solved problem. It requires deep domain knowledge combined with machine learning intuition. I remember spending three weeks just testing different feature combinations before we saw consistent improvements over our baseline.
Another critical lesson involves normalization. Market data is non-stationary—volatility today can be ten times what it was last month. Without proper normalization, the agent's policy becomes brittle. We implemented adaptive scaling where features are normalized using rolling windows, but even this had pitfalls. During the COVID-19 market crash of March 2020, our rolling windows captured such extreme values that subsequent data looked "calm" by comparison, causing the agent to become overly aggressive. We had to introduce regime-aware normalization that adjusts its duration based on volatility estimates.
To give you a sense of the complexity here, consider the dimensionality problem. A mid-frequency execution algorithm might observe order book levels at 10 price points on each side, plus trade indicators, plus meta-parameters. That is easily 50+ dimensions. RL algorithms generally struggle with high-dimensional state spaces due to the curse of dimensionality. We addressed this with autoencoders pre-trained on historical data to compress the state into a 10-dimensional latent representation. This trick alone improved convergence speed by roughly 40% in our backtests.
--- ##Reward Design: What Matters Most
If state representation is the eyes of the agent, reward design is its soul. In execution algorithms, the objective is multi-faceted: minimize market impact, control timing risk, avoid adverse selection, and sometimes even signal intent to the market. Designing a reward function that captures all these dimensions is both an art and a science.
The most common approach is to use implementation shortfall as the primary reward. Implementation shortfall measures the difference between the execution price and a benchmark price (typically the arrival price). A negative shortfall means we bought cheaper than the market at order arrival—good. But this naive reward ignores numerous practical considerations. For example, if the market moves favorably after we start trading, aggressive execution might capture that move better than passive execution, yet a pure implementation shortfall reward does not distinguish between skill and luck.
At ORIGINALGO, we modified our reward to include a risk-adjusted implementation shortfall. We add a penalty term proportional to the variance of execution prices, effectively making the agent risk-averse. This is crucial because, in practice, fund managers care not just about average execution quality but about consistency. A strategy that occasionally saves 10 basis points but occasionally loses 50 basis points is not acceptable. We also incorporate a penalty for extreme inventory positions, which prevents the agent from hoarding trades and waiting for perfect conditions that never come.
Research by Moody and Saffell (2001) on reinforcement learning in finance laid the groundwork for reward shaping in trading systems. More recently, Deng et al. (2022) proposed a multi-objective reward structure where the agent learns to trade off between immediate market impact and long-term market dynamics. This resonated with our experience; we found that a single scalar reward often leads to degenerate policies. For instance, an agent might learn to trade extremely slowly to minimize impact, but this exposes it to massive timing risk if the market moves against the order.
We experimented with Pareto-optimal reward formulations where we train multiple agents, each with different weights on impact vs. risk. This gives the trader a menu of execution styles to choose from based on their current tolerance. One client—a large asset manager rebalancing a pension fund—consistently chose the high-risk version because they had high signal conviction and wanted speed. Another client—a mutual fund dealing with daily flows—preferred the conservative version. This customization would have been impossible with traditional rule-based algorithms.
An often-overlooked aspect is the temporal credit assignment problem. When you execute an order over 30 minutes, the reward from a trade made at minute 1 depends on the state created by trades made at minute 29? Actually, no—later trades are influenced by earlier ones. If you trade aggressively early, you might move the market and make later trades more expensive. The agent needs to attribute reward correctly across time steps. We use eligibility traces with TD(λ) methods to address this, which proved far more stable than standard Q-learning in live trading.
--- ##Market Impact Modeling: The Core Challenge
Market impact is the elephant in the room for any execution algorithm. When you trade, you move prices. That impact decays over time but never fully disappears. Traditional methods model impact using formulas like the Almgren-Chriss model, which assumes a linear relationship between trading rate and impact. In reality, impact is convex, time-varying, and path-dependent. Reinforcement learning offers a way to learn impact dynamics directly from data, without assuming a parametric form.
We at ORIGINALGO spent considerable effort building a custom simulation environment that embeds a learnable impact model. The idea is simple: use historical data to train a neural network that predicts price changes given a sequence of trades. This network then becomes the "market simulator" for training our RL agent. This approach has two advantages. First, the agent learns in an environment that reflects actual market behavior, not stylized assumptions. Second, as we collect more data, we can update the impact model, allowing the agent to adapt to regime changes.
A paper by Cont and Kukanov (2017) provided theoretical underpinnings for this approach, showing that impact evolves as a multivariate Hawkes process with cross-excitation between trades and orders. We found this framework particularly useful for modeling the "echo" of large trades—how one aggressive order can trigger a cascade of subsequent orders from other market participants. Our impact model includes a self-exciting component that captures this phenomenon. In backtests, ignoring this cascade effect led to 15-20% underestimation of total impact for large orders.
Here is a real case from our experience. We were executing a large sell order for a mid-cap stock during a period of low liquidity. Our traditional impact model predicted 35 basis points of total impact. The RL agent, trained on the neural impact model, predicted 52 basis points. We followed the RL agent's advice—it broke the order into much smaller pieces and used more passive orders. The actual impact turned out to be 48 basis points. My team was thrilled, but more importantly, we saw a consistent 25% reduction in estimation error across a portfolio of 200 stocks over three months.
The key insight here is that impact is not stationary. It depends on who is on the other side of the trade. During earnings announcements, impact increases dramatically because informed traders are active. During lunch hours in Tokyo or London, impact decreases because liquidity providers are less risk-averse. Our RL agent learned these patterns implicitly, without being told about time zones or corporate events. This is the power of data-driven impact modeling.
One challenge we faced is that impact models trained on historical data might reflect past market conditions that no longer hold. For example, after the introduction of the Maker-Taker fee structure on certain exchanges, impact dynamics shifted significantly. Our RL agent initially overestimated impact because its training data predated the fee change. We now incorporate online learning techniques where the impact model updates continuously, and the RL agent retrains periodically. This adds complexity but is essential for robustness.
--- ##Exploration vs. Exploitation: Walking the Tightrope
In RL, the exploration-exploitation dilemma is the constant companion of every practitioner. The agent must balance acting optimally based on current knowledge (exploitation) versus trying new actions to discover potentially better strategies (exploration). In execution algorithms, this dilemma takes on heightened importance because every action has real financial consequences. Too much exploration, and you lose client money. Too little, and you never improve.
Our early approach used epsilon-greedy exploration, where the agent takes a random action with probability epsilon. This was catastrophic. On one occasion, an agent decided to "experiment" with an extremely aggressive order during a low-liquidity period, causing a price spike that cost us over $50,000 in slippage on a single order. The client was not amused. This incident taught us that naive exploration strategies are unacceptable in production trading systems.
We transitioned to parameter-space exploration and Thompson sampling. In parameter-space exploration, we add noise to the policy network's parameters rather than to the actions directly. This yields more coherent exploration—the agent might consistently trade slightly more aggressively but without erratic single-step experiments. Thompson sampling involves maintaining a distribution over optimal actions and sampling from it, which naturally balances exploration and exploitation based on uncertainty. A Bayesian neural network provides the uncertainty estimates needed for this approach.
A fascinating research direction from Bellemare et al. (2016) on "Unifying Count-Based Exploration and Intrinsic Motivation" introduced the concept of exploration bonuses based on novelty. We adapted this for execution: the agent receives an additional reward for visiting states that are rarely observed, such as extreme order book imbalances. This encourages the agent to learn about edge cases without directly penalizing poor execution. In practice, this prevented the agent from becoming overly conservative in normal conditions while ensuring it had experience handling stress scenarios.
Our current setup uses a three-stage approach. First, we pre-train the agent in a simulated environment with extensive exploration. Second, we fine-tune in a live environment with very conservative exploration (using Thompson sampling with a low temperature). Third, we run the agent in pure exploitation mode but maintain a shadow policy that explores on a small percentage of orders (less than 5%). This gives us continuous learning without exposing clients to excessive risk. I often joke that this is like training a pilot: you crash a thousand times in the simulator, then fly carefully with passengers on board.
One personal reflection: the exploration problem is fundamentally harder in financial markets than in games or robotics because the environment is not stationary. What was exploratory yesterday might be safe today, or vice versa. We have started using meta-learning techniques that allow the agent to adapt its exploration strategy based on recent market volatility. This is still experimental, but early results show that agents with adaptive exploration outperform those with fixed strategies by about 8-10% in terms of Sharpe ratio of execution performance.
--- ##Multi-Agent and Adversarial Settings
Financial markets are not single-agent environments. Every trade you make interacts with other market participants—liquidity providers, other algorithmic traders, high-frequency firms, and human investors. For realistic execution, we must consider multi-agent dynamics. If multiple trading desks run similar RL-based algorithms, they might collectively harm each other by competing for the same liquidity.
This problem became real for us when we deployed our execution algorithm across multiple client accounts trading the same stock simultaneously. Each agent was optimizing its own execution, but together they were creating an aggregate impact that none had anticipated. One day, two clients both had large sell orders in Apple stock. Our two agents independently decided to trade aggressively, and the combined selling pressure drove the price down by 15 basis points more than expected. The clients saved money individually, but collectively they paid more due to the interaction effects.
We addressed this by implementing a joint optimization framework where a central coordinator dispatches orders to different client agents. Each agent sends its desired trading trajectory to the coordinator, which then resolves conflicts by prioritizing orders based on urgency and size. This is essentially a multi-agent RL problem with a shared reward function. Research by Bu et al. (2022) on "Mean-Field Multi-Agent Reinforcement Learning for Execution" provided a scalable framework where agents consider the average behavior of other agents rather than explicit coordination. We found this approach more scalable than full centralization.
There is also the adversarial dimension: other market participants might learn to detect and front-run our algorithms. If a high-frequency firm identifies the signature of our RL agent, they might trade ahead of it, capturing the price movement we cause. We have observed this phenomenon in our data. Orders executed with high regularity—same size, same timing—are more likely to be detected. Our RL agent learned spontaneously to randomize its trading patterns, making it harder to predict. This emergent behavior was entirely unprogrammed; the agent discovered that unpredictability reduces adverse selection.
A paper by Spooner et al. (2018) on "Adversarial Reinforcement Learning for Limit Order Book Trading" formalized this idea, showing that agents trained with an adversarial component (a separate model trying to predict their actions) develop more robust strategies. We implemented a version where a discriminator network attempts to classify whether a sequence of trades originated from our agent or from random market activity. The executor then receives a penalty if the discriminator correctly identifies its trades. This game-theoretic approach has significantly improved our resilience against front-running.
Looking ahead, I believe the most exciting research in this area lies in cooperative-competitive multi-agent RL, where agents learn to collaborate with other non-adversarial algorithms while competing with adversarial ones. For example, multiple execution algorithms from different firms could theoretically form a "trading coalition" that aggregates passive orders to reduce market impact for all participants. This is speculative but not science fiction—I have already seen preliminary work from researchers at J.P. Morgan exploring this direction.
--- ##Real-Time Adaptation and Transfer Learning
Financial markets evolve continuously. A model trained on 2022 data may fail in 2024 because of regulatory changes, new market participants, or shifts in macroeconomic conditions. Real-time adaptation is not optional; it is a requirement for production systems. This is where reinforcement learning's online learning capability becomes invaluable, but implementing it safely is non-trivial.
At ORIGINALGO, we use a continual learning architecture where the agent updates its policy incrementally after each trading day. We fine-tune using recent experience weighted by a forgetting factor that emphasizes the last 20 trading days. This ensures the agent adapts to current conditions without catastrophic forgetting of general trading skills. We also maintain a rolling validation set to detect performance degradation. If the agent's execution quality drops below a threshold, we automatically revert to a baseline risk-controlled policy.
.png)
A specific challenge we encountered was concept drift during the transition from the COVID-19 crisis to the post-crisis recovery. Our agent had learned aggressive strategies that worked well in volatile markets, but as volatility subsided in mid-2021, those strategies caused excessive market impact. The agent initially resisted adapting because the old policy had higher expected reward in the training distribution. We had to implement adaptive learning rates that increase when the agent's prediction errors grow, forcing faster adaptation to regime changes.
Transfer learning offers another powerful tool for real-time adaptation. We pre-train a generalist agent on a diverse set of stocks, then fine-tune it for specific stocks. The pre-training phase learns universal features like order book dynamics and impact patterns, while fine-tuning captures stock-specific characteristics like typical volume profile or spread width. This dramatically reduces the data needed for each new stock. In practice, our generalist agent required only two weeks of fine-tuning data to match the performance of a specialist agent trained on three months of data for the same stock.
Research by Ganesh et al. (2021) on "Adaptive Reinforcement Learning for Real-Time Bidding" inspired our approach to fine-tuning. They used a two-stage process: shared representation learning followed by task-specific adaptation. We adapted this to execution by having a shared embedding layer that captures market-wide features, with stock-specific output layers. This architecture allows us to deploy a single model across hundreds of stocks while maintaining high performance per stock. Our deployment time for a new stock dropped from two months to one week.
One practical lesson: real-time adaptation must be monitored by human traders. We learned this the hard way when an agent started adapting to a data feed anomaly that showed artificially narrow spreads. The agent became overly passive, resulting in incomplete fills. Human oversight caught this within 30 minutes, but it highlighted the need for guardrails. We now have a monitoring dashboard that flags unusual changes in the agent's policy (e.g., sudden shifts in aggressiveness) and alerts the trading desk. I strongly believe that human-in-the-loop adaptation is essential for financial RL systems, at least for the foreseeable future.
--- ##Simulation Fidelity and Backtesting Pitfalls
Any RL practitioner in finance will tell you: backtesting is a minefield. The gap between simulated performance and live trading performance can be enormous, and execution algorithms are particularly susceptible to overfitting to simulation artifacts. Building a high-fidelity simulation environment is arguably the most important engineering challenge in this field.
Our first simulation environment was naive: we assumed that trades could be executed at the quoted bid-ask price without considering queue dynamics. The RL agent learned to place limit orders and expect immediate fills. In live trading, these limit orders would sit in the queue while the market moved, resulting in terrible execution. We had to incorporate a queue position model that simulates the probability of fill given the order size, crowd depth, and cancellation rates. This alone reduced the simulation-realistic gap by over 50%.
A particularly insidious pitfall is look-ahead bias. In simulation, the agent has access to future price information if the data is not carefully handled. For example, if you train using historical data where you know the next minute's volatility, the agent can learn to wait for calm periods. In live trading, it has no such knowledge. We addressed this by strictly simulating only past information—the agent never sees data it would not have at decision time. This sounds obvious but is easily violated when preprocessing data.
We also implement counterfactual reasoning to validate simulation fidelity. If the agent chooses action A in simulation, we compare the simulated outcome to what actually happened when a similar action was taken in live trading under similar conditions. Persistent deviations indicate that the simulation is missing important factors. For instance, we discovered that our simulation failed to capture the impact of regulatory announcements that occur at fixed times. The agent had learned to reduce trading intensity around announcement times in simulation, but in reality, the timing varied. Once we added a learnable event detector to the simulation, the fidelity improved markedly.
A paper by Coletta et al. (2022) on "Towards Realistic Market Simulators for Reinforcement Learning" proposed using generative adversarial networks to create synthetic order book data that captures complex dependencies. We tested this but found that GANs struggle to preserve the temporal structure of order books. Instead, we use a Bayesian bootstrap approach where we resample historical episodes with replacement, creating thousands of slightly different market scenarios that preserve the statistical properties of the original data.
My personal experience with backtesting pitfalls taught me humility. We once had an agent that showed 30% improvement over VWAP in backtest. In live trading, it showed only 8% improvement. After months of investigation, we found the culprit: our simulation assumed that all trades execute instantly at the mid-price, ignoring the spread crossing cost. The agent exploited this by frequently switching between buy and sell orders (a non-sensical behavior for a single-direction execution) because the simulation didn't penalize it. Once we added realistic spread costs, the agent's performance dropped to 12% improvement in backtest, and the live results aligned much better. This experience taught me to trust no backtest without understanding its assumptions.
--- ## Conclusion: The Road Ahead for RL in Execution Reinforcement learning for execution algorithms represents a paradigm shift from rule-based to data-driven trading. Throughout this article, I have explored seven critical aspects: state representation, reward design, market impact modeling, exploration strategies, multi-agent dynamics, real-time adaptation, and simulation fidelity. The common thread is that RL offers a flexible, adaptive framework that can capture complex market dynamics which are impossible to specify manually. However, the practical implementation is fraught with challenges that require deep domain knowledge, rigorous testing, and continuous monitoring. The financial industry is at an inflection point. Major banks and hedge funds are increasingly adopting RL for execution, driven by the realization that traditional algorithms leave significant alpha on the table. A 2023 study by McKinsey estimated that RL-based execution can improve performance by 10-20% across major asset classes, representing billions in potential savings annually for institutional investors. Yet adoption remains limited by the technical complexity and the shortage of practitioners who understand both finance and RL. Looking forward, I believe the most impactful developments will come from three directions. First, foundation models for market dynamics—large-scale pre-trained models that capture universal trading patterns, similar to how large language models capture language patterns. These could be fine-tuned for specific execution tasks with minimal data. Second, safe RL methodologies that provide formal guarantees about worst-case performance, making the technology palatable to risk-averse institutional investors. Third, human-AI collaboration frameworks where the algorithm suggests actions and the trader oversees, combining machine efficiency with human judgment. At ORIGINALGO TECH CO., LIMITED, we have already started moving in these directions. Our R&D team is working on a proprietary foundation model trained on over 10 years of order book data across 5,000 stocks. We are also developing safety layers that use control barrier functions to prevent the agent from taking actions that exceed predefined risk limits. And we have deployed a human-in-the-loop system where traders provide feedback on the agent's performance, which is then used to refine the reward function. These are early steps, but they point to a future where RL execution becomes as standard as VWAP or TWAP is today. I will end with a personal thought. When I joined ORIGINALGO three years ago, I was skeptical about RL for production trading. The research literature was promising, but the gap between academic papers and live trading seemed vast. Today, I am a convert. Not because RL is a silver bullet—it is not, and anyone who tells you otherwise is selling something—but because it offers a fundamentally different mindset: instead of fighting against market complexity by adding more rules, we embrace it and let the data speak. That, in my view, is the only sustainable path forward for execution algorithms in an increasingly complex financial world. --- ## ORIGINALGO TECH CO., LIMITED's Insights on Reinforcement Learning for Execution Algorithms At ORIGINALGO TECH CO., LIMITED, we have spent over three years integrating reinforcement learning into our execution technology stack, and our perspective is grounded in both technical rigor and practical experience. We believe that RL for execution is not merely a technological upgrade but a fundamental rethinking of how trading systems interact with financial markets. The key insight we have gained is that the value of RL lies not in replacing human expertise but in augmenting it. Our most successful deployments have been those where traders and algorithms work as partners—the algorithm handles the complex, high-dimensional optimization while traders oversee strategic decisions and handle exceptions. We have also learned that safety and interpretability are non-negotiable; no amount of performance improvement justifies a black-box system that can behave unpredictably. To this end, we have invested heavily in explainable AI techniques that allow our clients to understand why the algorithm made certain decisions. Finally, we believe that the future belongs to collaborative, adaptive, and safe RL systems that can operate across multiple asset classes and market regimes. We are actively researching meta-learning approaches that allow rapid adaptation to new conditions without extensive retraining, and we invite our peers in the industry to join us in advancing this exciting frontier.