The check we thought we had
Most bank statements print a running balance on every row. That gives you an obvious and satisfying check: take any row's balance, subtract the previous row's, and the result should equal that row's amount. Run it down the whole document and a misread number lights up on the exact row it occurred.
We had that. It worked. It caught real errors, it pointed at specific rows, and it made us confident — and the confidence was the problem, because we had quietly extended what it proved from "the amounts on these rows are consistent" to "nothing is missing from this statement".
Those are very different claims, and the second one is false. This article is the list of things that followed from believing it, including one check that was structurally incapable of ever reporting a problem.
Two gates, and the blind spot between them
A statement supports two independent arithmetic checks, and their blind spots are in different places. That fact is the whole article compressed into a paragraph.
The balance chain compares each row to its predecessor. It is excellent at catching a single misread amount on a row that carries a balance, and it localises the error precisely.
The totals identity checks that the opening balance plus every transaction equals the printed closing balance. It is a statement about the document as a whole rather than about any row.
The chain's blind spot is severe and was invisible to us: if a row is dropped entirely, the surviving rows either side are compared directly to each other. Their printed balances differ by whatever they differ by, and nothing in the comparison knows a transaction was meant to sit between them. The chain reports a clean pass over a statement missing a dozen rows.
The totals identity catches exactly that, because a dropped row leaves the sum short. We did not have it. Everything below follows from that gap.
Mistake one: treating a chain as completeness
The chain check answers a local question — is this row consistent with the one before it — and we were reading a global answer out of it. Every individual comparison passing does not mean the set of rows is complete, because the comparisons are only ever between rows that survived.
The failure mode this permits is the worst one in document extraction: rows lost at a page boundary. Repeated headers, carried-forward subtotals and real transactions all sit within a couple of centimetres of each other, and a reader that drops the first rows of a page produces a shorter statement that is internally perfectly consistent.
So we built the totals gate: opening balance plus every extracted transaction, compared to the printed closing balance, to the cent. It is a small piece of code and it is the single most valuable check in the pipeline, because it is the only one that has an opinion about rows that are not there.
Both checks now run. Neither replaces the other, and the general lesson generalises well beyond us: two cheap checks with different blind spots beat one clever check with a hole in it.
Mistake two: a check that could never fail
This is the one that is genuinely embarrassing, and the one most worth publishing because it is easy to write without noticing.
Some documents print no closing balance. We had a fallback for that: derive it by adding the transactions to the opening balance. Sensible enough for display purposes — a user wants to see a closing figure.
But that derived figure was then used as the target of the completeness check. So the check was asking whether the opening balance plus the transactions equalled the opening balance plus the transactions. It passed unconditionally, on every document, forever. It could not have failed if the extraction had dropped half the statement.
Worse than useless, because it reported success. A document with no arithmetic backstop at all was showing the same green result as one that had genuinely been verified.
The fix was to delete the fallback entirely. If a document prints no closing balance, we now report that the identity could not be evaluated. That is a worse-looking output and an honest one, and it prompts the correct response — sample the rows, because nothing here has been proved.
| Check | What input makes it fail? | Verdict |
|---|---|---|
| Balance chain | An amount misread on a row carrying a balance | A real check |
| Totals identity | A row dropped, invented or misread | A real check |
| Derived closing balance | Nothing whatsoever | Not a check |
The question that finds these
Mistake three: totals we invented
Our exports added a TOTAL row at the bottom of grids where the source document had no total. It seemed helpful. Someone opening a spreadsheet usually wants a sum.
It is fabrication, and a user pointed it out before we did. A spreadsheet exported from a bank statement carries an implicit claim that its contents came from the statement. A row that says TOTAL, in a document that never printed one, breaks that claim in a way nobody has any reason to suspect.
And it is actively dangerous, because the sum is computed from the same extraction that might have dropped a row. It will always agree with the data — including agreeing with the error. Someone checking that the spreadsheet adds up gets a confirmation that means nothing, generated by the thing being checked.
We removed invented totals from the verbatim grid and the statement sheet. If the document printed a total, it appears because it is data. If it did not, there is no total. A missing sum makes a person compute one and think about it; a fabricated one gets trusted.
Mistake four: a repair that was really tuning
Once you have a check that fails, the immediate temptation is to fix what it caught. And there is a legitimate version: a common reason a statement does not balance is that a summary row — a page subtotal, a carried-forward figure — was read as a transaction. Removing it is a genuine correction.
The danger is the shape of that operation. A system that adjusts extracted data until a gap gets smaller is not fixing errors, it is removing evidence. The output then reconciles and contains whatever change made the arithmetic work, which is strictly worse than an honest failure — a caught error converted into a silent one.
So the rule we settled on is narrow and absolute: a repair is kept only if the document's own printed figures then hold exactly. Drop the candidate rows, and if the totals reconcile to the cent against the bank's own opening and closing balances, the hypothesis is confirmed by evidence that existed before we touched anything. If the gap merely narrows, the change is discarded.
We also limited how hard it is allowed to try. Two principled attempts — first rows that are labelled as summaries, then all rows that have neither a date nor a balance — and then it stops. We deliberately do not search for whichever subset of rows makes the total work, because with enough rows some combination always will, by coincidence. A search permitted to keep going will eventually manufacture a passing result.
In practice this works well. One statement in our test set was out by 1,284.40 and reconciled to exactly zero once a labelled summary row was excluded — the document proved the fix. Others reduced their gap under various hypotheses and were rejected, which is the rule doing its job.
Mistake five: a tick we could not justify
Some statements cover several accounts under a single header — a current account, a savings account and a card, each with its own running balance, all in one document with one opening and one closing figure at the top.
The totals identity cannot be evaluated on such a document. One opening and closing pair cannot span several balance chains, so summing every transaction and comparing against that pair produces a number describing nothing. We were producing a result for it anyway.
Now the check stands down and reports that it could not be evaluated. This looks like a downgrade — a document that used to show a verdict now shows an admission — and it is a substantial improvement, because the previous verdict was noise presented as fact.
It also prompts the right action. "Could not be evaluated because this document covers multiple accounts" tells a user exactly what to do: separate by account and check each chain independently. A green tick told them nothing and implied everything.
We think this is the most important principle in the whole list. A check that reports success when it did not actually execute is worse than no check at all, because it converts uncertainty into confidence without adding any information.
Mistake six: a gap treated as a warning
Our validator scaled the size of a difference against the statement's turnover. A gap that was small relative to total activity produced a warning; a large one produced an error. The reasoning was that a tiny discrepancy on a busy account is less alarming than a large one on a quiet one.
That reasoning is wrong, and it took building the totals gate to see why. The identity is exact. Either the opening balance plus the transactions equals the closing balance or it does not, and a gap of any size means something is missing or misread. Relative size describes how much activity surrounded the error, not how serious it is.
In fact the relationship often runs the other way. A gap of a few cents is more likely to be a systematic problem — a rounding approach, a sign convention, a value date — than a one-off, and dismissing it because it is small relative to turnover is exactly how a systematic issue survives for a year.
A failing identity is now an error, full stop, with the gap reported in currency and the rows most likely implicated listed alongside it. The user decides whether it matters; we do not decide it for them by scaling it against something irrelevant.
What we found when the check finally ran
The interesting part of building a check that can fail is that it does.
Most statements reconciled immediately, which was the expected and reassuring result. A minority did not. Of those, roughly a third were repairable under the exact-or-nothing rule — almost all of them summary rows that had been read as transactions, where removing the row made the document's own arithmetic hold perfectly.
The remaining ones were genuine extraction errors that no arithmetic can recover: rows misread in ways that changed the total, or content that was never captured. That is worth stating plainly rather than rounding off — for those documents, we cannot repair the problem, only report it.
We should be precise about what those proportions describe: they are shares of the statements that failed the check, not of all statements. The point is not a failure rate. It is that a check which can fail tells you something, and one that cannot tells you nothing while feeling identical.
The ones we cannot fix
For the statements where the identity fails and no proved repair exists, the temptation is obvious: find something plausible to adjust so the number goes green.
We do not, and the reasoning is the same as with the invented totals. A document that quietly does not add up, presented as though it does, is worse for the user than one that says loudly that it does not. The first produces a ledger with a hidden error; the second produces ten minutes of annoyance and a correct outcome.
So the output is the gap in currency, the rows the checks implicate, and the ability to correct them directly — with the checks re-running on the corrections and the export built from the corrected version. Someone who knows the document resolves most of these in minutes, because they can see immediately that a row is a subtotal or that a number lost a digit.
That division of labour is, we think, the right one. Arithmetic identifies where the problem is with certainty; a person resolves what it is. Neither is good at the other's job.
Three designs we considered and rejected
Between finding the gap and shipping the totals gate we looked at several alternatives, and the reasons for rejecting them are more useful than the design we chose.
A tolerance band. Accept a difference below some threshold as immaterial. It is superficially attractive because a few cents feels like noise. But a tolerance is a decision that some errors do not count, taken by us on behalf of a user who does not know it was taken — and it is precisely the small systematic differences, from a rounding approach or a sign convention, that a band would swallow indefinitely. The identity is exact, so we report it exactly and let the user judge materiality.
A learned anomaly detector. Train something to recognise statements that look wrong. This has genuine appeal for the fields arithmetic cannot reach, and it is the wrong tool for completeness, because it would replace a proof with an estimate. A model that says a document is probably fine is strictly weaker than arithmetic that says the totals hold to the cent, and it cannot explain itself to an accountant.
A confidence-weighted score.Combine the checks into a single number. We rejected this for the reason running through the whole article: compressing several different questions into one figure destroys exactly the information the user needs. "Completeness proved, dates unverified, could not evaluate on this multi-account document" is three separate facts, and there is no honest way to average them.
What all three have in common is that they would have made the output look better and mean less. That trade is available at every point in this kind of system, and it is usually the wrong one.
The principles that came out of it
A check must be able to fail. If you cannot describe the input that would make it report a problem, it is decoration. Ask it of every check you have; some will not survive the question.
Never test a value against itself. A derived figure cannot be the standard for checking the thing it was derived from. This sounds too obvious to state and is exactly what we shipped.
Never output what was not in the source. No invented totals, no computed closing balances presented as though printed. If it was not in the document, it does not appear as though it was.
Repairs must be proved, not tuned.Exact reconciliation against the document's own figures, or discard the change. A narrowed gap is not evidence.
Standing down beats a false pass. When a check cannot run, say so. Reporting success without executing is the most damaging thing a validation layer can do.
Exact checks are exact. Do not scale a binary result against a magnitude. It either holds or it does not.
| Principle | The failure it prevents |
|---|---|
| A check must be able to fail | Decoration reported as verification |
| Never test a value against itself | A circular check that always passes |
| Never output what was not in the source | Fabricated totals trusted downstream |
| Repairs proved, not tuned | A caught error turned silent |
| Standing down beats a false pass | Confidence manufactured from nothing |
| Exact checks are exact | A real gap dismissed as immaterial |
The whole thing in a table
| What was wrong | Why it happened | What we do now |
|---|---|---|
| Chain check treated as completeness | It caught real errors, so we over-trusted it | Totals identity runs alongside it |
| Closing balance derived, then checked against | A display fallback leaked into validation | Report 'could not be evaluated' |
| TOTAL rows added to exports | Convenience; a user flagged it | Only totals printed in the source appear |
| Repair risked tuning a gap smaller | Fixing what the new check caught | Exact reconciliation or discard the change |
| Unbounded search for a balancing subset | More repairs looked better | Two principled attempts, then stop |
| Result shown for multi-account documents | The check ran and produced a number | Stands down; reports not evaluable |
| Gap scaled by turnover into a warning | Small differences felt less serious | Any gap is an error, reported in currency |
If you are building something like this
Test with broken documents. A validation suite run only against correct inputs has tested nothing. Remove a row, alter an amount, flip a sign, delete a page, and confirm each check reports what it should. Every mistake in this article would have surfaced immediately under that discipline.
Keep display logic away from validation logic. Our derived closing balance was a perfectly reasonable display convenience that became a validation input. Separate the two, or make derived values structurally unusable as check targets.
Make "not evaluated" a first-class result. If your check has only pass and fail, it will be forced to produce one of them in situations where neither is true. Three outcomes, not two.
Prefer several cheap checks to one clever one. The reason we were exposed was that everything rested on a single mechanism. Two checks with different blind spots would have caught the page-boundary case years earlier.
And know the boundary of what you have even after all this. The arithmetic covers amounts and completeness and nothing else — dates, payees and references pass every one of these checks untouched, which is the subject of the errors that pass every check.
Key takeaways
A row-to-row balance chain does not prove completeness. When a row is dropped, the surviving neighbours are compared directly and agree perfectly, so the chain closes over the gap. The totals identity — opening plus every transaction against the printed closing balance — is what catches it, and both should run.
The most instructive bug was a check that could never fail, because the closing balance it tested against was derived from the very transactions being tested. The question that finds this class of problem is simple: what input would make this check report something?
And the principle we would keep above all the others: when a check cannot run, say so. Standing down on a multi-account document is a worse-looking result and a far more honest one than a verdict nothing supports.
Frequently asked questions
See what the checks say about your statement
Both gates run on every document. You get the result, the gap in currency when it fails, the rows implicated, and an explicit note when a document makes the check impossible to evaluate.