Confidence is not proof
Ask most document extraction tools how accurate they are, and the honest answer underneath the marketing page is a number the model gave itself — a confidence score, averaged across a batch of fields, presented as if it were a measurement of correctness. It isn't. Confidence is a model's estimate of its own certainty, and an estimate is not the same claim as a proof.
This page is the benchmark we actually run, on every document, and it's deliberately not a single flattering percentage. It's a test: does the arithmetic on the statement itself hold. Opening balance, plus every transaction extracted, against the closing balance the bank printed. If it holds, to the cent, something real has been proven. If it doesn't, the gap is real too, and it names exactly where.
Why a model's own confidence is the wrong number
Ask an extraction model to rate its own confidence on a field it read correctly, and it tends to answer somewhere in a narrow, high band — not because the document was genuinely that certain, but because that's roughly where models tend to answer when asked to rate themselves on exact matches. The number moves a little from document to document. It does not move nearly as much as the actual correctness of the extraction does.
The practical result is a confidence score that looks meaningful — it's a number, it has decimal places, it sits in a chart — while barely distinguishing a statement that reconciled perfectly from one that's missing real rows. A score that can't tell those two documents apart isn't measuring accuracy in any sense a reader of a benchmark page actually cares about.
This isn't a criticism specific to any one model or vendor — it's a structural property of asking a system to grade its own work with no external check. The fix isn't a better-calibrated confidence prompt. It's replacing self-assessment with something the document itself can prove or disprove.
What “benchmark” means here
A benchmark most people picture is a fixed test set, run once, producing a headline number a vendor puts on a slide. That kind of benchmark tells you how a tool did on someone else's documents, under someone else's conditions, at some point in the past. It says nothing directly about your document, today.
The benchmark on this page is a different kind: a repeatable, arithmetic test that runs on whatever document you actually upload. It doesn't require trusting a number we published about a test set you'll never see. It requires trusting arithmetic, on your own statement, which you can check by hand if you want to.
| Kind of benchmark | What it actually tells you |
|---|---|
| A published leaderboard percentage | How a tool did on a fixed set of documents you cannot see or verify |
| A model's self-reported confidence | How certain the model says it is, which correlates only weakly with correctness |
| An arithmetic completeness proof | Whether this specific document's extraction is provably complete, right now |
The actual test
A bank statement carries its own proof, printed right on it, and almost nobody uses it: the opening balance, the closing balance, and every transaction in between. Add the opening balance to the sum of every transaction extracted. If the result equals the printed closing balance, to the cent, then nothing was dropped, duplicated, or misread badly enough to move the total — the extraction is provably complete, not just plausible.
Alongside that total check, a second, row-level check runs the same proof one step at a time: each row's own printed balance must equal the previous row's balance plus that row's amount. Where a statement prints a balance on every row, this pinpoints exactly which row breaks the chain, not just that something, somewhere, is wrong.
Neither check needs a labelled answer key, a human reviewer, or a test set. The statement supplies the answer; the extraction either matches it or it doesn't. That's what makes this a proof rather than an opinion — it can be run on a document nobody at FlowParse has ever seen, and it still means something.
What a real benchmark has to measure
Field-level accuracy — was this date read correctly, was this amount read correctly — is the number most vendors report, and it's a real, useful measurement of one specific thing. What it structurally cannot measure is absence: a row that was never extracted at all doesn't show up as a wrong field, because there's no field there to be wrong. It simply isn't counted.
| Measurement | Catches | Misses |
|---|---|---|
| Field-level accuracy | A wrong date, a mistyped amount, a misread description | A row that was never extracted at all |
| Model confidence | Nothing directly — it's the model's own opinion of itself | Everything a confidence score wasn't calibrated to notice |
| Balance-gate completeness proof | Anything, of any kind, that changes the total by more than a rounding cent | The specific field-level nature of the error, without a second pass |
The honest answer isn't that one measurement replaces the other — it's that a benchmark claiming to measure accuracy without also measuring completeness is only telling half the story, and it's the half that's easier to make look good.
A statement that looked 99% right
The clearest illustration of the gap between accuracy and completeness is a real shape of failure, not a hypothetical: a multi-page statement where every extracted row is genuinely correct — right date, right amount, right description — and one row, sitting exactly on a page break, was never extracted at all.
| What a field-accuracy check sees | What the balance gate sees |
|---|---|
| Every returned field is correct — no visible defect | Opening + transactions does not equal the printed closing balance |
| No low-confidence flag, because no field was uncertain | A specific gap, sized exactly to the missing row |
| A clean-looking export, ready to ship | A document held back for review before it ships |
Field accuracy alone reports this document as essentially perfect. It has no way to notice a row that simply isn't there. The balance gate reports it as failing, points at the exact page and the exact size of the gap, and holds it back from export until the gap is resolved — which is the entire reason the check exists.
How to run it on your own document
Upload a real bank statement
PDF, scan or photo — ideally one that prints both an opening and a closing balance.
Extraction runs
Every transaction read, with dates, amounts and the running balance where the statement prints one.
The balance gate runs automatically
Opening plus transactions checked against the closing balance, and each row against the one before it.
You see the score and the specific rows
A 0-100 quality score, and, if anything failed, exactly which rows and by how much.
No signup is required to try this once, and nothing about the test changes based on who's running it — the same arithmetic applies whether it's your first upload or your thousandth.
Against a general AI model, directly
A common real-world comparison is pasting a bank statement into a general chat model — ChatGPT, Claude, Gemini — and asking it to extract the transactions. It's a genuinely capable thing for these models to attempt, and on a short, clean statement the result often looks entirely plausible.
What that workflow doesn't include, by default, is an arithmetic proof step — nothing checks the model's own output against the statement's printed balances before you see it, and a long statement can silently exceed what the model actually processed in full. The full comparison, including where a general model is genuinely a reasonable choice, is in FlowParse vs ChatGPT, Claude and Gemini for bank statements.
Against a traditional OCR platform
Older, template-based OCR platforms take a different approach entirely — a fixed template maps specific coordinates on a page to specific fields, which works well until a bank changes its layout by even a few pixels, at which point the template silently misreads or drops data with no indication anything went wrong.
A modern AI-native reader like FlowParse doesn't depend on a fixed template, which handles layout variation better — but layout independence alone still isn't a completeness proof. Whichever extraction approach is behind a tool, the honest question is the same: does anything check the result against the document's own arithmetic before it's trusted.
Reading the score
The number FlowParse actually shows is a deterministic 0-100 Quality Score, computed from the checks that passed and failed — not a model's self-reported confidence, and not a fixed value averaged from a test set you'll never see. A clean, fully reconciling statement reaches 100. Every failed check deducts a fixed, documented amount.
| Score band | What it typically means |
|---|---|
| 95-100 | Clean document — checks passed, balance reconciles |
| 80-95 | A couple of warnings worth a quick look, not confirmed errors |
| 50-80 | Real problems flagged, specific rows named |
| Below 50 | Serious, usually a genuine misread or missing data — reviewed before export |
A warning is a flag worth a second look, not proof of an error — the checker is deliberately strict about small mismatches, so a document with a single flagged field can read 93% while remaining entirely correct. Only rows the balance gate names as genuinely failing represent a proven problem.
What happens when a page fails the test
A failed balance check isn't the end of the process — it's a signal that triggers a more careful, more expensive re-read of the specific page in question, rather than shipping a document the arithmetic itself says is incomplete. That escalation is bounded by a real cost budget, which is stated plainly rather than hidden: some documents are held for a heavier pass, and some are shipped flagged rather than silently.
What never happens is the test being quietly dropped because it produced an inconvenient answer. The score you see reflects the arithmetic, not a decision about how the arithmetic ought to be presented.
Statements with more than one account
A single PDF often carries several accounts in sequence — a current account, then a savings account, each restarting its own running balance. Chaining them together as if they were one continuous sequence would make every account's opening row look like a broken transaction, which is a false failure, not a real one.
The benchmark accounts for this by grouping rows into per-account chains before running the check, so a correct multi-account statement is verified per account rather than penalised for a boundary the check itself introduced. The reported result names how many accounts were found and verifies each on its own terms.
When the document can't check itself
Not every document supports this proof. A card statement or an export with no printed running balance genuinely cannot be checked this way — there's no opening-plus-transactions arithmetic to run when the document never printed the numbers that arithmetic needs.
In that case the honest answer is “not verifiable,” shown plainly rather than scored as if the check had passed. Not verifiable and reconciled are different claims, and collapsing them into one green checkmark would be exactly the kind of quiet overstatement this whole page argues against.
What this benchmark doesn't prove
Doesn't prove every individual field is correct
Two equal and opposite errors could, in principle, cancel out and still balance — the row-level check exists precisely to catch most of that class, but it isn't a mathematical impossibility.
Doesn't apply to documents with no running balance
A card statement or an export with no printed opening/closing figures is marked not verifiable, not passed by default.
Doesn't measure how a tool performed on someone else's documents
It measures your document, right now — which is a narrower but more honest claim than a fixed test-set percentage.
Doesn't replace human review of what the extraction actually returned
A reconciling total tells you nothing is missing; it doesn't tell you every description or category is exactly how you'd have written it.
A benchmark that claimed to prove everything would be making a bigger claim than any arithmetic test actually supports. Stating the limits plainly is what keeps the parts it does prove trustworthy.
Why there's no single headline percentage on this page
It would be easy to put one number at the top of this page — “99.x% accurate” — and it would almost certainly get more clicks than a page explaining a methodology. It would also be a claim we couldn't stand behind the way this page can, because a single number requires a fixed test set, and any fixed test set is a claim about documents you'll never see, measured under conditions you can't verify.
The arithmetic proof doesn't have that problem, because it isn't a claim about a hidden test set — it's a claim about your document, checkable the moment you upload it. That's a narrower promise than a headline percentage. It's also one we can actually keep.
Who this is for
Procurement and IT teams
Evaluating document extraction vendors and wanting a test they can actually run, not a slide to take on faith.
Finance teams and accountants
Who want to know a converted statement is complete before it feeds into the books, not just that it looks right.
Developers integrating an extraction API
Who need a score they can gate a pipeline on automatically, not a confidence number that barely moves.
Anyone who has been burned by a 'looked fine' export
Where a missing row surfaced weeks later as a reconciliation that wouldn't tie.
If you're specifically evaluating several vendors side by side, the practical method for doing that fairly is in how to benchmark document extraction tools, and the evaluation criteria a procurement team typically needs are in vendor evaluation for procurement teams.
Privacy
Uploads go over TLS, encrypted end to end.
Processing runs on EU-hosted infrastructure.
Original documents are deleted immediately after extraction.
Documents are never used to train AI models.
Full details are on the security page.