High-Frequency Data Cleaning and Normalisation

Introduction: The Unseen Engine of Modern Finance

In the world of high-frequency trading (HFT), quantitative analysis, and AI-driven finance, we often marvel at the sophisticated algorithms, the lightning-fast execution speeds, and the complex predictive models. Yet, beneath this glittering surface of technological prowess lies a less glamorous, but fundamentally critical, foundation: the meticulous, often grueling, process of high-frequency data cleaning and normalisation. This article isn't about the flashy trading strategies; it's about the unsung hero that makes them all possible. At ORIGINALGO TECH CO., LIMITED, where our daily grind revolves around building robust financial data strategies and AI systems, we've learned a hard truth: a model is only as good as the data it consumes. Garbage in, gospel out is a dangerous fantasy. The reality is garbage in, garbage out, amplified at nanosecond speeds. High-frequency data—tick-by-tick trade and quote information, order book updates, and real-time economic feeds—is inherently messy, incomplete, and fraught with anomalies. Before any analysis can begin, before any AI model can be trained, this torrent of raw information must be transformed into a clean, consistent, and reliable resource. This process is not merely a technical prelude; it is the very bedrock of accuracy, profitability, and risk management in today's data-centric financial markets.

The stakes are astronomically high. Consider a personal experience from our early days developing a market-making signal. We had a beautiful mean-reversion model that showed incredible backtested profits. We deployed it with confidence, only to watch in horror as it executed a series of loss-making trades. The culprit? A flaw in our data cleaning logic failed to properly handle "stub quotes"—non-binding, far-off-the-market price quotes often used by exchanges to fulfill regulatory obligations. Our model, seeing a sudden, massive but entirely artificial "spread" in the order book, interpreted it as a massive arbitrage opportunity and leapt in. We were trading against ghosts. That painful, expensive lesson burned into our philosophy: the data pipeline is not an IT problem; it is the first and most important line of defense in algorithmic strategy. This article will delve deep into the multifaceted discipline of high-frequency data cleaning and normalisation, exploring its key challenges, methodologies, and profound implications from the trenches of financial technology development.

The Anatomy of Raw Tick Data

To understand cleaning, one must first appreciate the chaos of the source. Raw high-frequency tick data is a relentless, unstructured stream of events. Each line represents a trade execution, a quote update (bid or ask change), or an order book modification. It arrives from multiple venues—exchanges, dark pools, electronic communication networks (ECNs)—each with its own formatting quirks, timestamp granularity (nanoseconds, microseconds), and reporting conventions. The first layer of complexity is asynchronous multi-source ingestion. Data from Exchange A might be delayed by a few milliseconds compared to Exchange B due to geographic latency or internal processing queues. A "simultaneous" price movement across two exchanges will appear at different times in your raw feed. Simply merging these streams chronologically without understanding the source latency profiles creates a false sequence of events, a mis-telling of market history that can distort any subsequent analysis.

Furthermore, the data is riddled with genuine market microstructure "noise" that isn't an error but must be understood. You have bid-ask bounce, where prices naturally oscillate between the buy and sell price without a change in fundamental value. You have flash crashes and mini-flash events—brief, extreme price movements that may be real trades (perhaps due to a large "market sell" order) or errors. Then there are the outright errors: fat-finger trades (e.g., a trader mistakenly entering an order at $100 instead of $10), duplicate transmissions where the same trade is reported multiple times, and timestamp reversals, where later events are logged with an earlier timestamp due to system clock drift or processing lag. At ORIGINALGO, we once spent a week debugging a volatility calculation that seemed inexplicably high. The issue traced back to a single day where the data vendor's feed, due to an internal glitch, had repeated a 10-minute segment of data three times in the stream. The raw data "looked" continuous, but the repeated price jumps artificially inflated volatility metrics. Isolating this required building cross-day pattern recognition into our cleaning pipeline, a step beyond simple range checks.

Timestamp Synchronisation and Alignment

In high-frequency analysis, time is not just a dimension; it is the most precious and problematic variable. The core challenge of timestamp synchronisation is creating a consistent, trustworthy timeline across all data sources. Different exchanges timestamp events at different points in their internal workflow—when the order was received, when it was matched, or when it was reported. Network latency varies, and even your own servers' clocks can drift. The first step is often to normalize all timestamps to a single, high-precision time standard, like Coordinated Universal Time (UTC) with nanosecond resolution. But normalising the format is just the start.

The more profound task is temporal alignment for cross-sectional analysis. If you want to calculate a meaningful correlation between the price of Apple stock on NASDAQ and a related ETF on the NYSE, you need their price observations to be aligned in time. Do you snap both to a regular clock grid (e.g., every 100 milliseconds)? If so, what do you do if no trade occurred in that interval? Do you carry the last price forward? Use an interpolated bid-ask midpoint? The choice dramatically affects the resulting statistical properties. For order book analysis, alignment is even trickier. A snapshot of the "national best bid and offer" (NBBO) at any microsecond requires consolidating the top-of-book quotes from dozens of venues, all timestamped to that exact same moment. A misalignment of even a few microseconds can show a crossed market (where the bid exceeds the ask) that never truly existed, triggering faulty arbitrage signals. Our approach has evolved to use event-driven windows rather than rigid clock ticks, aligning data based on the sequence of significant events (like a trade or a large quote change) to preserve the true causal structure of the market, even if it makes the resulting time series irregularly spaced.

Filtering and Imputation of Missing Data

Missing data in a low-frequency context might mean a missing monthly data point. In high-frequency finance, "missing" can mean a gap of milliseconds, but in those milliseconds, a critical trade or quote update may have been lost. Data feeds drop packets. Exchange systems hiccup. The first line of defense is identifying the "unfillable" gaps. A short gap of a few milliseconds in a quote feed for a highly liquid instrument during active trading hours is suspicious and often indicates a data loss. A similar gap at 4:05 AM EST is likely just a period of no market activity. Distinguishing between the two requires understanding market hours, session breaks, and typical liquidity patterns.

Once a gap is identified, the decision is whether and how to impute. Imputation in a high-frequency context is a dangerous game. Simple linear interpolation of prices between two points assumes a constant, smooth price movement, which is almost never true—prices move in discrete jumps. A more sophisticated method might involve using a state-space model or leveraging correlated instruments to infer the likely price path, but this injects model assumptions into the raw data. At ORIGINALGO, our default principle is "when in doubt, leave it out and flag it" for strategies sensitive to completeness. For some analytical purposes, like calculating daily VWAP (Volume-Weighted Average Price), a few missing ticks may be tolerable. For an execution algorithm trying to minimize market impact, missing a large hidden-order trade could be catastrophic. We implement a tiered imputation strategy: for short, intra-millisecond gaps in liquid periods, we may carry the last known quote forward with a confidence flag. For longer gaps, we leave a clear NaN (Not a Number) marker and ensure our downstream models are robust to missing data, perhaps by switching to a coarser time scale for that period.

Outlier Detection: Signal vs. Noise

This is perhaps the most nuanced aspect of cleaning. An "outlier" is not necessarily an error; it might be the most important signal in your dataset—the proverbial needle in the haystack. A true market-moving event, like an earnings surprise or a central bank announcement, will cause a massive, rapid price change. This is valid data. A fat-finger trade or a system glitch that causes the same price spike is an error that must be removed or corrected. Telling them apart in real-time is fiendishly difficult. Simple statistical filters, like removing data points beyond X standard deviations from a rolling mean, are blunt instruments that will smooth away genuine volatility and flash crashes, which are themselves important phenomena to study.

Advanced outlier detection must be context-aware and multi-dimensional. It doesn't just look at price. It considers: 1. Volume: A price jump on a single 100-share trade is more suspect than the same jump on a 100,000-share block. 2. Order Book Context: Did the trade occur inside the prevailing bid-ask spread, or far outside it? A trade at the bid or ask is more credible. 3. Cross-Venue Validation: Did the same price movement occur simultaneously on other exchanges where the asset is traded? If only one venue shows a spike, it's likely an error on that venue. 4. Time-of-Day: A 10% spike in a major equity at 2:00 PM is different from the same spike at 3:59:59 PM (just before close) or in the after-hours session. We employ machine learning models trained on historical data labeled with known errors (like exchange-issued correction notices) to score the likelihood of a tick being an anomaly. This model considers dozens of features in real-time. It's not perfect, but it's a world away from simple Z-score filters. The key is to never delete "outliers" blindly; they are quarantined, reviewed, and either reinstated as valid or corrected based on a set of hierarchical rules, often involving looking for subsequent "cancel and correct" messages from the exchange.

Normalisation for Comparative Analysis

Cleaning ensures data is correct. Normalisation ensures it is comparable. This is crucial when you want to train an AI model on multiple assets or create a composite signal. The raw price of Apple ($~180) and Berkshire Hathaway Class A ($~600,000) are on completely different scales. A $1 move is negligible for BRK.A but significant for AAPL. The most common form is returns normalisation: converting prices into logarithmic returns (percentage changes). This transforms the data into a (more) stationary series with stable statistical properties, centered around zero. This is essential for feeding data into neural networks or clustering algorithms, as it prevents the model from being dominated by the absolute price level.

Beyond simple returns, there is volatility scaling. A 1% return on a calm day is different from a 1% return on a highly volatile day (like during an FOMC announcement). To compare signals across time and assets, we often normalize returns by a rolling measure of volatility, such as the 20-day exponential moving average of standard deviation. This creates a series of "risk-adjusted" moves. Another critical normalisation for high-frequency order book data is depth and spread scaling. The absolute bid-ask spread for a penny stock might be $0.01, and for a large-cap stock, it might also be $0.01. But the economic meaning is vastly different. Normalising the spread by the mid-price (creating a percentage spread) or by the average daily range allows for meaningful comparison of liquidity across instruments. At ORIGINALGO, when building a universal market-making model, we spent months designing a normalisation scheme that could handle everything from major FX pairs to small-cap equities. The solution was a pipeline that first cleaned the raw ticks, then constructed standardised features (like volatility-scaled returns, percentage spreads, and order book imbalance normalized by average daily volume), creating a homogeneous "feature space" where our AI could learn patterns applicable across the board.

Building a Robust, Auditable Pipeline

All these techniques are useless if they are not embedded in a production-grade, auditable data pipeline. This is where the rubber meets the road in financial data strategy. The pipeline cannot be a one-off Python script; it must be a version-controlled, modular, and monitored system. Each cleaning and normalisation step must be a discrete module with clear inputs, outputs, and parameters. This allows for reproducibility: you must be able to re-process historical data from two years ago with the exact same logic you used then to ensure your backtests remain consistent. Version control (like Git) for the pipeline code and its configuration files is non-negotiable.

Equally important is data lineage and audit logging. For every output tick, you should be able to trace its provenance: Which raw tick(s) did it come from? What cleaning rules were applied? Was it flagged as an outlier? Was it imputed? This traceability is critical for debugging (like our repeated data segment issue) and for regulatory compliance. Regulators increasingly want to understand the data journey that led to a trading decision. Furthermore, the pipeline must have real-time monitoring and alerting. Key metrics—like data latency, missing packet rates, outlier percentages, and the statistical properties of the cleaned output—should be continuously tracked. A sudden spike in the number of "corrected" trades or a drop in the correlation between cleaned data from two parallel sources should trigger an immediate alert. In our infrastructure, we treat the data pipeline as a mission-critical trading system, with the same redundancy and failover requirements. After all, if the pipeline breaks, every downstream model and trading strategy is flying blind.

The Human-in-the-Loop: Review and Governance

Despite all the automation, the human element remains vital. You cannot fully automate judgment. A regular review process must be in place where data scientists and quants examine samples of the cleaned data, particularly the records that were heavily altered—the outliers that were removed, the large gaps that were imputed. This review validates the automated rules and catches "unknown unknowns"—anomaly types the system hasn't been trained on. It's also essential for governance. Changes to the cleaning logic are not mere code updates; they are changes to the fundamental definition of your firm's "truth" dataset. Such changes should go through a formal review and approval process, much like a model validation for a trading strategy. What's the impact of tightening the outlier threshold? Will it change the historical performance profile of our strategies? These questions must be asked and answered before deployment.

This human oversight also extends to vendor management. When a data feed consistently shows issues, it's not just a technical problem; it's a commercial one. Having detailed, auditable logs of data quality issues provides concrete evidence for discussions with vendors about service level agreements (SLAs) and credits. I recall a protracted negotiation with a market data provider where we used our pipeline's audit logs to demonstrate a persistent 5-millisecond latency skew in their Asian session feed compared to their direct competitor. This wasn't a gut feeling; it was a chart of timestamp deltas across millions of events. That data-driven argument got the issue prioritized and resolved. The cleaning pipeline, therefore, becomes not just an analytical tool, but a key asset in managing the entire data supply chain.

Conclusion: The Strategic Imperative

High-frequency data cleaning and normalisation is far more than a technical preprocessing step. It is a core strategic discipline that directly determines the validity of research, the robustness of AI models, and the profitability and risk profile of automated trading systems. It demands a hybrid skillset: deep understanding of market microstructure, statistical rigor, software engineering excellence, and meticulous operational governance. The process is iterative and never truly "finished," as markets evolve, new instruments emerge, and novel forms of data errors inevitably appear.

Looking forward, the field will be shaped by several trends. The increasing use of alternative data (satellite imagery, social sentiment, credit card transactions) introduces new cleaning challenges with unstructured, noisy, and sparsely sampled data that must be fused with traditional market data. AI and machine learning will not only consume cleaned data but will also play a larger role in the cleaning process itself, with self-learning systems that adaptively identify new anomaly patterns. Furthermore, the rise of decentralized finance (DeFi) and on-chain data presents a fascinating new frontier, where the "tape" is theoretically immutable and transparent, but requires entirely new normalisation techniques to account for blockchain-specific artifacts like gas price fluctuations and miner extractable value (MEV). The firms that invest in building a deep, institutional competency in this unglamorous foundation will build a sustainable, defensible advantage. They will be the ones whose models see the market clearly, whose strategies are built on rock, not sand, and who can navigate the coming data deluge with confidence. In the race for alpha, the first and most important lap is run in the data preparation pipeline.

ORIGINALGO TECH CO., LIMITED's Perspective

At ORIGINALGO TECH CO., LIMITED, our journey in financial data strategy has cemented a fundamental belief: data cleaning and normalisation is not a cost center, but a primary value driver. It is the critical translation layer between the chaotic reality of market events and the structured world of quantitative models. Our experience building AI-driven trading and risk systems has taught us that the most elegant algorithm will fail if fed a distorted view of the world. Therefore, we treat our data pipeline as a first-class product—continuously investing in its resilience, intelligence, and transparency. We advocate for a philosophy of "defensive data consumption," where every piece of data is assumed guilty until proven clean through a rigorous, auditable process. Our forward-looking insight is that as AI becomes more autonomous, the integrity of the data substrate becomes even more paramount. We are moving beyond reactive cleaning towards predictive data quality management, where the system anticipates and mitigates potential data issues before they corrupt downstream processes. For us, mastering high-frequency data cleaning is synonymous with building trust in our technology, ensuring that our solutions are not just fast and smart, but fundamentally reliable and sound.

This in-depth article