Every vendor's number is unfalsifiable
Search for any document extraction tool and you'll find an accuracy percentage on the homepage, usually a confident one, and almost never with the test set, the ground truth, or the methodology attached. That's not necessarily dishonest — it's just a number you have no way to check, measured on documents you'll never see, under conditions nobody outside the vendor can verify.
This guide is the alternative: a method for testing accuracy yourself, on your own real documents, that produces a result you actually trust because you watched it happen. It doesn't require trusting anyone's claim — including ours.
None of what follows requires a data science background or special tooling. The method is deliberately simple enough to run with a spreadsheet and a stopwatch — the value comes from doing it consistently and on the right documents, not from anything sophisticated about the technique itself.
It also works regardless of which tools end up on your shortlist. Whether you're comparing purpose-built financial extraction tools, general-purpose document APIs, or a general AI chat model used casually, the same nine steps apply — real documents, a decision on what correct means made before testing, and a completeness check that doesn't depend on trusting anyone's claim.
Why a vendor's own benchmark can't be trusted blindly
It isn't that vendors lie about their numbers. It's that a benchmark run by the party being benchmarked has an unavoidable incentive problem — the test set gets chosen, tuned and reported by someone whose business depends on the result looking good. Even with entirely good intentions, that's a hard position to test objectively from.
| Who runs the test | What it tells you |
|---|---|
| The vendor, on their own test set | How the tool did on documents chosen and graded by the party selling it |
| A third-party benchmark site | How it did on a fixed, shared set — better, but still not your documents |
| You, on your own documents | Whether it works on the actual thing you'd be running through it every day |
This isn't unique to document extraction — it's the same reason a car's advertised fuel economy and a phone's advertised battery life rarely match real-world use exactly. The manufacturer isn't lying; the test conditions were simply chosen by the party with the most reason to want a good result. Document extraction accuracy claims have the same structural shape, and deserve the same healthy skepticism.
Three decisions before you start
What documents actually represent your real workload
Not your cleanest examples — the ones with the layouts, scan quality and formats you'll genuinely feed the tool every week.
What 'correct' means for your test
Field-level accuracy against a known answer, completeness against the document's own arithmetic, or both — fixed in advance.
Who runs the test, and whether they'll see the result before it's final
A test designed and scored by someone with a stake in a specific outcome tends to find what it's looking for.
The nine steps
Gather twenty to thirty real documents, including your worst ones
Actual documents from your own workflow — not clean demo PDFs — deliberately including your hardest scans and least standard layouts.
Decide what 'correct' means before you run anything
Field-level accuracy against a known answer, or completeness against the document's own arithmetic, or both — decided in advance so the test isn't graded after the fact to favour a result.
Run every tool on the exact same document set
The same documents, in the same order, with no cherry-picking a tool's easier subset after seeing how the others performed.
Test completeness using the document's own arithmetic
Where a document carries a self-contained proof — a bank statement's opening and closing balance, an invoice's line-item total — check the extracted output against it, no answer key required.
Deliberately break each tool's assumptions
A multi-account statement, a foreign currency, an unusual date format, a scanned photo taken at an angle — the cases that actually separate tools in practice.
Time the full workflow, not just the extraction call
Upload, extraction, review of anything flagged, and export — the real time cost includes all of it, not only the fastest-looking step.
Check how each tool reports a failure
A silent wrong answer is worse than a loud correct refusal — note whether a tool tells you when it's uncertain or simply guesses without saying so.
Test the export step against your real destination
Whatever your documents actually need to become — an accounting import, a spreadsheet, a downstream API — confirm the output actually works there, not just that a file was produced.
Run the whole test again on a second, independent document set
A single run can be a coincidence in either direction. A second set, ideally chosen by someone who didn't see the first result, confirms whether the difference is real.
Scoring a run without a labelled answer key
Building a ground-truth answer key by hand for every field on thirty documents is real work, and most teams evaluating a tool don't have days to spend on it. The good news is that a large share of what actually matters can be tested without one, using arithmetic the document itself already provides.
A bank statement prints an opening balance and a closing balance. Sum the extracted transactions, add the opening balance, and check the result against the printed closing balance — if it matches, the extraction is provably complete on that document, no answer key required. An invoice's line items should sum to its printed total; the same principle applies.
This doesn't replace field-level accuracy checking entirely — it won't catch a category that's simply wrong while the total still balances — but it catches a specific, dangerous class of failure (missing or duplicated rows) that field spot-checking routinely misses, and it does it for free.
Where a document doesn't carry this kind of built-in arithmetic — a card statement with no running balance, a contract with no numeric total — completeness has to be checked a different way: count the number of line items or transactions a careful manual read of the source finds, and compare it to how many the tool actually returned. It's slower than an arithmetic check, but it's the same underlying question, asked the only way that document allows.
A real proof of concept, run three ways
A finance team evaluating extraction tools for a switch away from manual entry ran the same thirty bank statements through three candidates: a general-purpose OCR API, a specialised financial extraction tool, and a general AI chat model given the same PDFs one at a time.
| Tool | Balanced (opening + txns = closing) | Needed manual correction |
|---|---|---|
| General OCR API | 22 of 30 | 8 statements needed re-checking against source |
| General AI chat model, one PDF at a time | 24 of 30 | Longer statements truncated without warning |
| Specialised financial extraction tool | 29 of 30 | 1 statement flagged and correctly held for review |
The most useful result wasn't the ranking itself — it was that the specialised tool's one failure came with a flag naming the exact problem, while the other two tools' failures were only discovered because the team happened to check the arithmetic afterward. Knowing you have a problem is worth almost as much as not having one.
It's worth noting what this test didn't settle. The general OCR API and the chat model each had genuine strengths outside this specific scenario — broader document-type coverage in the first case, flexibility for one-off questions in the second. A narrow benchmark on one document family tells you which tool fits that family best, not which tool is best overall, and a good evaluation keeps that distinction explicit rather than letting one result stand in for a broader claim it was never designed to support.
Common mistakes
Testing with clean, easy demo documents
Every reasonable tool handles a clean, well-formatted PDF well. The differences that matter only show up on the hard, real documents you actually receive.
Only checking field accuracy, never completeness
A tool can be excellent at reading correctly and still silently drop rows — field accuracy alone has no way to notice absence.
Letting each vendor pick their own test documents
This measures each vendor's best case, not a comparable result across tools.
Stopping the test at extraction, never checking export
A tool that extracts beautifully but produces a file your accounting software can't import hasn't actually solved your problem.
Running the test only once
A single run can be a coincidence. Extraction quality can vary meaningfully across documents even for the same tool.
Best practices
Test with your own documents, deliberately including the hardest ones you actually receive.
Decide what counts as correct before running any tool, not after seeing the results.
Use each document's own arithmetic as a free, objective completeness test wherever it's available.
Score how failures are reported, not just how rare they are.
Run the test at least twice, on independent document sets, before trusting the result.
Building a test set that won't flatter anyone
A good test set is boring to assemble and disproportionately valuable once it exists. Pull the last three months of whatever documents you actually process — not a curated highlight reel, the real inbox — and set aside twenty to thirty that cover your genuine range: your best-formatted statements, your worst scans, your most unusual layout, anything with more than one currency or account.
Resist the temptation to swap out a document because it seems “unfairly hard” for a tool — that document is exactly the reason you're testing. If it's part of your real workload, it belongs in the set regardless of which tool it happens to embarrass.
A useful sanity check once the set is assembled: if you showed it to someone on your team who hasn't been involved in the evaluation and asked whether it looks like a normal month of documents, the honest answer should be yes. A set that looks unusually clean, or unusually hard, in either direction has drifted from representing your real workload — and a benchmark built on a set that isn't representative answers a question you didn't actually ask.
The edge cases that actually matter
| Edge case | What it tests |
|---|---|
| A scanned or photographed document | Whether OCR quality holds up, not just clean-PDF text extraction |
| A statement with several accounts in one file | Whether the tool understands account boundaries or chains everything together incorrectly |
| An unusual date format or a foreign currency | Whether locale assumptions are hard-coded or genuinely handled |
| A document that prints a balance once per day, not per row | Whether completeness checks accumulate correctly across gaps |
| A genuinely long document | Whether length causes silent truncation, especially in a general chat-model workflow |
None of these are exotic scenarios — they're the ordinary shape of a real document workload. A tool that only gets tested on the easy 80% will have its weaknesses discovered by your business, in production, instead of by you, during the trial.
A practical way to prioritise, if time is short: rank your own document set by how often each edge case actually appears in your real inbox, and test in that order. A layout you see once a year deserves a note, not necessarily a full test cycle. A layout you see every week deserves to be the very first thing you check, because that's where a weakness will cost the most time if it turns out to exist.
Testing completeness, not just correctness
Field-level accuracy and completeness are genuinely different properties, and conflating them is the single most common gap in an informal benchmark. A tool can return every field it did extract with total correctness while quietly failing to extract several rows — every returned field passes a spot-check, and the missing rows are simply invisible to that check.
Testing completeness doesn't require sophisticated tooling — it requires picking documents that carry their own arithmetic and actually running the check, which most informal evaluations skip simply because nobody thought to ask the question.
It also changes how you should read a vendor's own accuracy claim once you understand the distinction. A claim like “99% field accuracy” is a real, checkable statement about individual fields, and it is silent, entirely, on completeness — a vendor reporting that number in good faith may simply never have measured the other question at all, not because they're hiding something, but because most published benchmarks in this space were never built to ask it.
Timing the whole workflow, not just the call
A tool's API response time is one number; the time your team actually spends per document is a different and more important one. It includes uploading, waiting for the result, reviewing anything flagged, correcting what needs correcting, and exporting — and a tool that's fast on the extraction call but produces output nobody trusts without a full manual re-check hasn't actually saved time.
Don't stop testing at extraction
Extraction is the step every vendor demos. The export is the step that determines whether the tool actually fits your workflow, and it's tested far less often — try exporting into whatever your real destination is, an accounting import format, a downstream API, a specific spreadsheet template, and confirm it actually works there rather than assuming a structured output is automatically compatible.
How failures are reported matters as much as how often
Two tools with an identical failure rate are not equivalent if one names the failure and the other doesn't. A tool that flags “this row is uncertain, please check it” hands you a short, specific review task. A tool that silently returns a plausible-looking wrong answer hands you nothing — you find out only when something downstream doesn't add up, usually much later and at a worse time to fix it.
When comparing tools, deliberately note not just whether each one failed on a given document, but whether it told you it had.
A scorecard template you can reuse
A scorecard doesn't need to be complicated to be useful. A handful of columns, filled in per tool per document, makes the comparison legible instead of a vague impression everyone remembers slightly differently a week later.
| Column | What it captures |
|---|---|
| Document and edge case tested | Which of your real documents, and which specific challenge it represents |
| Balanced against its own arithmetic | Pass or fail on the completeness test, where applicable |
| Flagged its own uncertainty | Whether the tool told you about a problem, or you had to find it yourself |
| Manual correction needed | What, specifically, had to be fixed by hand before the output was usable |
| Time from upload to usable export | The real workflow cost, not just the extraction call's response time |
Five columns, filled in consistently across every tool and every document, usually make the real difference between candidates obvious well before the last document in the set.
Keep the completed scorecard even after the decision is made. Six months in, when someone asks why a particular tool was chosen — or when the same evaluation needs repeating at renewal time — the raw scorecard answers the question directly, without anyone having to reconstruct the reasoning from memory or a slide deck that's since been lost.
Red flags in how a vendor talks about accuracy
A single percentage with no methodology attached
Accurate compared to what, measured how, on what documents — a number with none of those answered is closer to marketing than to evidence.
Reluctance to run a trial on your own documents
A tool confident in its own accuracy should welcome a test on real, hard cases rather than steering you toward its own demo set.
No distinction between confidence and correctness
A vendor that presents a model's self-reported confidence as if it were a measured accuracy figure is presenting an opinion as a fact.
No mention of what happens when extraction fails
Every extraction tool fails on some documents. One that doesn't discuss what a failure looks like hasn't been asked the harder question yet.
Why one test run is never enough
A single run, even a careful one, can be misleading in either direction — a tool might have an unusually good or unusually bad day on your particular thirty documents for reasons that have nothing to do with its typical performance. Running the same method again on a second, independent set of your real documents is what turns a promising first impression into a result you can actually act on.
This doesn't need to mean doubling the entire evaluation effort. A second run of ten to fifteen documents, assembled independently rather than reused from the first set, is usually enough to confirm or complicate a first impression without turning the evaluation into a second full project.
If a second run tells a materially different story than the first, that disagreement is itself useful information — it usually means the difference between the tools is smaller than either run alone suggested, which changes how much weight accuracy alone should carry in the final decision.
What to do with the result
A completed benchmark rarely produces a single obvious winner across every dimension — more often it produces a clear picture of trade-offs: one tool is more accurate on your hardest documents, another is meaningfully cheaper at your real volume, a third has an export format that fits your accounting system perfectly. Deciding which trade-off matters most is a business call the benchmark informs; it doesn't make the call for you.
For a structured way to weigh those trade-offs alongside accuracy — pricing, support, compliance, integration effort — see vendor evaluation for procurement teams. And if part of your test included a general AI chat model as a candidate, the specific gaps that tend to show up are detailed in FlowParse vs ChatGPT, Claude and Gemini for bank statements.
Whatever the outcome, resist the pressure to compress the whole result into a single winner for a slide. The trade-offs the benchmark surfaced are real information, and flattening them into one recommendation loses exactly the nuance that made running a careful test worth the effort in the first place.
Frequently asked questions
Include FlowParse in your test
Upload your hardest real statement and see the balance-gate result for yourself — no signup required.