Open source · local AI

I built an AI that checks whether digital products deliver what they promise.

A listing can say “48 pages”, “A4 and US Letter”, and “debt repayment worksheets.” PromiseDiff checks whether the delivered files can actually support those claims.

By Pavel Kazantsev2026-10-14 · 7 min read

A mismatch between a digital-product listing and its download turns into refunds, support messages, rough reviews, and the feeling that something obvious slipped through. PromiseDiff is a local-first pre-publish check for PDF sellers: paste the listing, upload the PDF or ZIP, and get back what the files support, contradict, or cannot prove.

PromiseDiff compares a product listing with delivered files and produces an evidence-checked report — The aim is not confidence. It is an answer that stays attached to its evidence.

Missing evidence is not a contradiction

This is the rule I cared about most. A scanned document can have no extractable text. A claim can name an ambiguous file. A model can point to a quote that is not actually on the page. None of those proves the seller wrong.

Result	Meaning
`SUPPORTED`	The supplied evidence supports the claim.
`CONTRADICTED`	A reproducible check found a conflicting fact.
`UNVERIFIED`	The files do not establish the claim either way.
`OUT_OF_SCOPE`	The MVP cannot test this kind of claim.

Matrix explaining supported, contradicted, unverified and out of scope verdicts — Unverified prevents absence of proof from becoming a false accusation.

Two evidence paths

Page counts, dimensions, and file inventory are measurements, so PyMuPDF checks them directly. Semantic promises need interpretation: a “debt repayment worksheet” may be called a “Loan Balance Reduction Tracker” in the PDF. For that path, local Gemma through Ollama reads relevant excerpts.

Listing claims and delivered files move through deterministic and semantic verification paths — Measurements and semantic matching meet at one evidence check.

A model answer is not evidence

The model can suggest a result, but cannot certify its own work. A semantic finding becomes supported only when the filename exists, the page exists, and the exact quote occurs in extracted text. Otherwise it remains unverified.

A model citation must pass filename, page and exact quote checks before it becomes a supported finding — The model proposes; the document decides.

A real mismatch in the demo

The included Budget Planner really has 48 A4 pages and the expected planner content. Its sample listing also promises a US Letter edition. The file does not contain one, so PromiseDiff confirms the page count and flags the missing edition with the measurement retained in the report.

Short PromiseDiff walkthrough showing the product verification flow — A compact walkthrough recorded and cut with ProgressCut.

I recorded the walkthrough with ProgressCut, my local-first tool for turning a longer screen recording into a short visual story. They make a useful pair: PromiseDiff checks the release before it ships; ProgressCut helps explain it after it is ready.

Untrusted uploads need a boundary

PDFs and ZIPs arrive untrusted. The backend caps upload size, PDF count, page count, expanded ZIP size, and extracted text; it rejects traversal paths, duplicate entries, symlinks, encryption, nested archives, and suspicious compression.

Security checks between an untrusted PDF or ZIP upload and the analysis path — A public deployment should still isolate PDF processing in a resource-limited worker.

A small benchmark, including the miss

Ten independently labelled synthetic cases run through the same local API as the UI. The runner saves raw responses and timings rather than manufacturing a headline score afterwards.

Benchmark summary showing nine exact verdict labels out of ten, with one semantic paraphrase unresolved — Nine of ten labels matched. The one miss is the semantic paraphrase case.

Metric	Result	Boundary
Exact-label accuracy	90% (9/10)	Tiny synthetic evaluation.
Contradiction precision / recall	100% (2/2)	Measured facts only.
Supported recall	75% (3/4)	One paraphrase stayed unverified.
Median API latency	2.11 s	Local Gemma on this machine.

The miss matters: “debt repayment worksheet” was not promoted to support for “Loan Balance Reduction Tracker” because the evidence was not strong enough. Better paraphrase retrieval is next; the rule stays the same—when the file cannot establish a claim, say so.