My ML Feature Used the Wrong Unit. The Threshold Was 400× Too Large.

The historical model and the live validator both calculated large_share, the fraction of sweep volume in unusually large trades. Their definitions had diverged. The live code used a percentile of per-second net aggressive volume as the cutoff for an individual trade. On the affected symbols, that cutoff was 163–423 times too large.

The public audit package includes the minimal reproducible fixture, the feature-contract checks, and the recorded aggregate results used here. It does not include the underlying market-data archive.

The first failure was a unit mismatch

The historical feature used the 75th percentile of individual trade sizes. The live validator took the 75th percentile of nav_abs, a per-second aggregate. The two thresholds measure different quantities.

import numpy as np
large_thr = np.percentile(np.abs(dev["nav_abs"]), 75)
# nav_abs: per-second net aggressive volume

large_share = volume_of_trades_at_or_above(large_thr) / total_volume
# threshold is applied to individual trades

That made the test almost impossible to pass. The root-cause audit found that only 0.001–0.008% of trades cleared the live cutoff, leaving large_share near zero in 97% or more of live events. The repair takes the percentile from individual trade quantities and refuses to score a symbol without a matching calibration artifact.

Three other feature contracts also differed

The feature-parity audit found three more mismatches between the training and live contracts. They matter because a model can appear to have a distribution problem when its inputs are simply calculated differently.

The large_share defect caused most of the false positives. In the first, analytical triage of 14 October, fixing it alone reduced selected candidates from 40 to 5. Applying all four corrections left 2. These are diagnostic counts from before the October raw-trade archive was recovered; two feature values were approximated in that first pass.

Scope of the replay

We subsequently recovered the Binance Futures aggTrades archive for 6–14 October. To avoid mixing validator sessions, the raw reconstruction uses the V1 live session for 6–13 October (13,786 candidates) and the V2 full baseline for 14 October (895). It rebuilt the four affected feature families at the recorded timestamps for 14,681 of 14,682 candidates. One LDOUSDT detection second was absent from the archive.

The frozen artifacts reproduced the recorded model scores with a mean absolute difference of 1.04×10⁻⁶. The repaired features selected 52 candidates. Their raw five-second Binance markout averaged +8.67 bps; after a 10 bps round-trip fee assumption, it was −1.33 bps. That is enough to reject a deployment claim.

Tests that would have caught this before a model score

A KS test is useful once two systems agree on the feature definition. It does not establish whether they used the same unit, aggregation window, or direction convention. Keep a small set of recorded raw inputs and run both implementations against them.

for fixture in recorded_sweeps:
    historical = batch_features(fixture.trades, fixture.direction)
    live = streaming_features(fixture.trades, fixture.direction)

    assert historical.large_share == live.large_share
    assert historical.run_frac == live.run_frac
    assert historical.pre_vel == live.pre_vel
    assert historical.trailing_nav == live.trailing_nav

This compares feature values before scores, signals, P&L, or a drift statistic enter the picture. Record the calibration unit, symbol, source period, and input hash beside each threshold.