216 Tests Passed. My Python Validator Still Returned a False PASS.

[1] and [true] are different JSON values. My Python comparator said they were equal.

{
  "state": "PASS",
  "deterministic_parity_verified": true,
  "findings": []
}

The tool compares a historical replay with a live-style decision pipeline, looking for the first recorded difference in features, model scores, decisions, configuration, or input evidence. A strict PASS is supposed to be its strongest result. This one was wrong.

The false PASS, reduced to five lines

baseline = {"payload": [1]}
candidate = {"payload": [True]}

[1] == [True]             # True
{"x": 1} == {"x": 1.0}    # True

Python is behaving lawfully. The bug was mine: scalar values went through type-aware logic, but nested lists and dictionaries eventually reached ordinary Python equality. Convenient application behavior became an invalid equivalence relation for a parity checker.

Before and after

$ hfm-parity diff --baseline examples/nested_values/baseline.jsonl \    --candidate examples/nested_values/candidate.jsonl
FAIL: mode=strict; not-verified; aligned=1; findings=1

The fixture is deliberately synthetic. It proves a regression boundary, not a historical market measurement or a claim about trading performance.

Why 216 tests did not catch it

The candidate had 216 Python tests when a second audit supplied this counterexample. Scalar integer semantics had been fixed. Clean installs, CLI and MCP demos, Ruff, mypy, and the frontend build passed. The gap was structural: the suite tested type semantics for scalars, not at every JSON leaf.

More happy-path tests would not have made that contract explicit. If a verifier can return PASS, every structural depth needs values that must fail.

What the tool is intended to compare

Backtests and live pipelines often disagree for annoyingly ordinary reasons: a feature was defined differently, a model version changed, timestamps use different clocks, or an event came from a different source. The final decision is usually too late in the chain to explain much.

Each JSONL trace records run metadata plus event-level fields: event ID, stage, features, timestamps, input hash, model score, decision, and optional state hash. The engine aligns event.id + stage, then returns PASS, FAIL, NOT_COMPARABLE, or UNVERIFIABLE. Strict mode is for two runs that claim to reproduce the same thing; regression mode can compare deliberate old-versus-new changes, but never certifies strict parity.

The first audit found other failures too

The first independent audit began with 87 passing tests and found nine problems. These three are the easiest to reproduce.

1. Large integers were converted to floats

The first helper turned numbers into float values before comparison. Past 2**53, IEEE-754 doubles cannot represent every integer, so the initial engine could call distinct values equal and return a verified strict PASS. Integers are now compared exactly; an integer and a float are a type mismatch even when their printed values look equal.

2. Matching event IDs could hide a different market

Two traces can both include evt-17, while one comes from BTC and the other from ETH. Strict mode now treats declared symbol, venue, or data_sourcemismatches as NOT_COMPARABLE. Missing provenance produces UNVERIFIABLE, not a polite-looking agreement.

3. The MCP output limit was per value, not per response

The first formatter shortened nested values one at a time, then assembled the response. A roughly 10 MB input could yield about 17.7 MB from one comparison call. The current implementation caps the whole response, preserves totals, labels truncation, and drops detail in a deterministic order.

The repair: compare the tree, not its container

The comparator now walks lists and objects recursively. At every leaf it compares JSON kind and value: null, boolean, integer, float, string, list, object, and missing are distinct. Numeric tolerance is available only for float-versus-float comparison.

difference(left, right, path):
  if json_kind(left) != json_kind(right):
      return type_mismatch(path)
  if both are lists or objects:
      recurse into children
  if both are floats:
      apply tolerance
  otherwise:
      compare exact values

That is intentionally stricter than Python equality. In a validator, 1,1.0, and true can encode materially different pipeline behavior.

PASS needs evidence, not just equal rows

The earlier audit found other problems: large-integer precision, missing provenance, ambiguous event order, and an MCP response cap that did not cap the final payload. They point to the same design rule: equality is not proof of comparable execution.

Strict mode is for two runs that claim to reproduce the same execution. It requires compatible metadata and input evidence. Deliberate old-versus-new comparisons use regression mode; they can be useful, but never certify deterministic parity. CSV Quick Compare follows the same rule: equal rows stay UNVERIFIABLE without provenance.

Try the public fixture

git clone https://github.com/icesurf666/hfm-parity-lab.git
cd hfm-parity-lab
python -m venv .venv && source .venv/bin/activate
pip install -e .
hfm-parity diff --baseline examples/nested_values/baseline.jsonl \  --candidate examples/nested_values/candidate.jsonl

The fixture, regression tests, and source are public. The current suite has 242 tests; that is regression coverage for known failures, not proof that no unknown failures remain.