How to Handle Out-of-Order Ticks and Data Delays in Real-Time Stock Market Data with Python

For developers using stock APIs to build market dashboards or quantitative trading systems, receiving Tick data does not necessarily mean the data is ready to use. In actual operation, network latency, message queues, and asynchronous task scheduling can cause the order in which market data arrives to differ from the order in which events occurred. If a program updates the latest price directly based on arrival order, an older quote may overwrite a newer one, potentially affecting candlestick generation and strategy backtesting.

The key to solving this problem is to distinguish between event time and arrival time, then determine how to handle out-of-order and late-arriving data based on the application's real-time requirements.

Why Does Out-of-Order Data Occur in Real-Time Stock Market Data?

Suppose a stock generates three consecutive Ticks with event timestamps of 10:00:00.100, 10:00:00.200, and 10:00:00.300. However, due to network conditions and program scheduling, the application may receive them in the order of the third, first, and second Tick.

If the price is updated every time a message arrives, the final displayed price may no longer represent the latest market quote.

Therefore, when processing real-time stock market data, two timestamps need to be distinguished:

• Event Time: The time associated with the market event represented by a Tick.
• Arrival Time: The time when the application actually receives the message.

Arrival time is useful for monitoring data latency, while event time is more suitable for determining the chronological order of market events. The two should not be treated as interchangeable.

For example, AllTick's WebSocket Tick data includes fields such as code, seq, tick_time, price, and volume. These fields can be used to identify securities, record market timestamps, and process trade data. For specific field descriptions, refer to the AllTick Tick Data Subscription Documentation.

Set Up a Bounded Reordering Buffer in Python

Waiting until all messages have arrived before sorting them can prevent a real-time market data application from updating promptly. A more practical approach is to use a bounded waiting window (reorder buffer). This temporarily stores Ticks that arrive within a short period, sorts them by event time, and then passes them to downstream processing.

The following is a simplified Python implementation that uses heapq to maintain an independent priority queue for each stock.

import heapq
from collections import defaultdict


class TickReorder:
    def __init__(self, lateness_ms=500):
        self.lateness_ms = lateness_ms
        self.max_seen = {}
        self.buffers = defaultdict(list)
        self.counter = 0

    @staticmethod
    def to_ms(value):
        # Supports common Unix timestamps in seconds or milliseconds
        ts = int(value)
        return ts if ts >= 10**12 else ts * 1000

    def flush(self, symbol, force=False):
        ready = []
        watermark = (
            float("inf")
            if force
            else self.max_seen[symbol] - self.lateness_ms
        )
        heap = self.buffers[symbol]

        while heap and heap[0][0] <= watermark:
            ready.append(heapq.heappop(heap)[3])

        return ready

    def push(self, tick):
        symbol = tick["code"]
        ts = self.to_ms(tick["tick_time"])
        seq = int(tick.get("seq", 0))

        if (
            symbol in self.max_seen
            and ts < self.max_seen[symbol] - self.lateness_ms
        ):
            return self.flush(symbol), tick

        self.max_seen[symbol] = max(
            self.max_seen.get(symbol, ts), ts
        )

        heapq.heappush(
            self.buffers[symbol],
            (ts, seq, self.counter, tick)
        )
        self.counter += 1

        return self.flush(symbol), None

Here, the tolerance window is set to 500 milliseconds. The program records the maximum event time observed for each stock and subtracts the tolerance window to calculate the current watermark. When a Tick's event time is earlier than or equal to the watermark, the program removes it from the queue and passes it to downstream processing.

seq is used as a secondary sorting key when multiple Ticks have the same event timestamp. However, it should not be treated as a global sequence number across different securities without verification. The timestamp conversion in this code also only covers common Unix timestamp formats in seconds and milliseconds. During actual integration, the conversion logic should be based on the verified API data format.

When the second value returned by push() is not None, it means the current Tick has exceeded the tolerance window. It should not simply be treated as invalid data. One option is to save it in the raw market data storage for historical data correction or separate analysis. When the program is shutting down or the buffer needs to be drained, the remaining data can be retrieved by calling flush(symbol, force=True).

Data Latency and Out-of-Order Ticks Require Different Handling

The 500-millisecond window is only an example and is not suitable for every market data system. A larger window increases the likelihood of restoring message order, but it also increases the waiting time for real-time processing.

For a market dashboard, the primary concern is usually displaying the latest price. In this case, the application can prioritize maintaining the latest quote and use event timestamps to determine whether an incoming message is newer than the current state. This prevents delayed data from overwriting newer quotes.

For real-time candlestick generation, trading volume aggregation, and strategy backtesting, event order is more important. Data should be sorted using a bounded buffer before aggregation. For data arriving after the watermark, business rules should determine whether previously generated candlesticks need to be corrected. Not all late-arriving Ticks should be discarded simply to maintain real-time performance.

It is also important to monitor latency between event time and arrival time, the proportion of out-of-order messages, buffer length, and the number of late-arriving Ticks. If latency continues to increase, simply expanding the buffer window will only increase waiting time. It will not solve persistent message accumulation upstream.

What to Consider When Connecting a Stock API

AllTick provides REST and WebSocket interfaces for accessing market data. WebSocket is more suitable for continuously updated real-time stock market data, while REST is suitable for retrieving quotes or historical data on demand. Developers can refer to the AllTick API Documentation for current interfaces and integration requirements.

Before deploying the system, developers should use actual streaming data to verify the unit of tick_time, the ordering semantics of seq, and the data handling process after a disconnection and reconnection. It is also important to note that the order in which an application receives messages only reflects their arrival sequence. It does not, by itself, prove that the upstream data provider has missed any market updates.

A reliable processing workflow for a stock market data system is not simply about sorting Ticks. Instead, it requires a clear trade-off between low latency, event ordering, and data completeness based on the application's actual requirements.