Engineering What Survives
Writing
Reproducible experiments about AI, markets and probability—and the useful information hiding inside failed results.
My ML Feature Used the Wrong Unit. The Threshold Was 400× Too Large.
A live ML pipeline compared individual trade sizes with a per-second NAV threshold. large_share collapsed toward zero; the audit found three more train/serve feature-contract bugs.
We Tested 12,720 Exit Rules. The Result Wasn’t Validated.
An HFM exit rule reported +25.98 bps per event. An audit found session concentration, reused validation, and missing joint outputs. A practical research postmortem.
I Removed Duplicate Sources From My RAG. The First Comparison Lied.
A RAG audit found that source deduplication also changed the evidence budget. A fixed-budget control corrected the claim; a 15-chain pilot validates the pipeline, not effectiveness.
I Tested 7 Crypto Models. The Best Signal Failed the Cost Test.
Seven crypto models, six walk-forward folds and a backtest audit: why positive predictive signal does not guarantee profits after modeled costs.
216 Tests Passed. My Python Validator Still Returned a False PASS.
A nested JSON type mismatch passed a Python trace validator after 216 tests. The repair is recursive and type-aware; equal CSV rows still need provenance.
I Let an LLM Agent Search for Trading Strategies. A Simple Heuristic Beat It.
I gave DeepSeek V3 forty chances to find a profitable trading strategy. A simple heuristic found +52% net. The agent surfaced three silent bugs, then optimized its way to +123.7% — a number that didn't survive scrutiny.
14,399 Events Looked Significant. 365 Trading Days Did Not.
A crypto breakout simulation produced a naive t-statistic of +8.20. Equal-weighting its 365 trading days produced −1.66 and exposed one exceptional day behind the positive mean.
I Built an AI That Checks Whether Digital Products Deliver What They Promise
A local-first QA tool compares a digital-product listing with the PDF or ZIP a buyer receives. It measures facts directly, validates model citations, and leaves uncertainty as unverified.
58 Trading Hypotheses. Most Died. Here's the Framework That Killed Them.
48 of 58 hypotheses were rejected. The five-gate kill pipeline that caught them: costs, robustness, null benchmarks, a sealed holdout, and a contamination audit. Each gate is a different question. Run the cheap ones first.
Green Tests, Broken Architecture: Four Models Review TypeScript Diffs
Four models reviewed 13 small TypeScript changes that could pass functional tests while violating architecture, recovery or filesystem invariants. Near-identical scores hid three different failure modes.
I Killed My Electron App Twice to Test Crash Recovery
SIGKILL during capture and real FFmpeg encoding exposed what a local-first recorder needs to preserve — and a video-duration bug the test caught.
From 10,000 Frames to 30 Moments: Building a Visual Story Compressor
Two hours of coding generates ~14,000 screenshots. Cutting that to 30 moments — no ML, no cloud — takes five passes over the same data, each killing a different category of noise.
Three Crypto Edges. What Survived the Audit?
Costs, parameter sensitivity, and a funding-carry result that changed sign on reproduction.
My Football Model Passed Validation. A Five-Check Audit Killed It.
Val away CLV was +152 bps, confidence interval excluded zero. Then I ran a five-check forensic audit: opening-line baseline, Brier skill, feature contamination, incremental CLV, and season stability. Four failed.
My Crypto Backtest Had a Sharpe of 13.4. Then I Added Trading Costs
The same cross-sectional reversal signal fell from a pre-transaction-cost Sharpe of 13.44 to a net Sharpe of -40.13. Its trading bill was four times its gross PnL.