Invoice parser in Python — why templates collapse, and what to check instead

Where everyone starts, and why it works at first

The first version is keyword anchoring: find the word *Total*, take the money-shaped token on the same line, move on. It is quick, it is readable, and on the first three suppliers it is right every time — which is precisely what makes it dangerous, because the approach looks validated when it has only been lucky.

The second version adds positional rules: this supplier's invoice number is always in the top-right block, that one's total sits in the bottom-right box. Now you have a template per supplier, a dictionary of coordinates, and something that resembles a small internal product.

The third version is where the trouble starts, because the templates begin to conflict, new suppliers arrive weekly, and every fix risks a regression on a layout you cannot easily test. The code below is the honest version of stage one, with its ceiling written into the comments.

# The approach everyone starts with, and its ceiling.
import pdfplumber, re

TOTAL_LABELS = ("amount due", "total due", "grand total", "balance due", "total")
MONEY = re.compile(r"[-(]?[\d.,]{1,15}[)]?$")

def find_total(page):
    """Look for a money-shaped word on the same visual line as a total-ish label."""
    words = page.extract_words()
    lines = {}
    for w in words:
        lines.setdefault(round(w["top"] / 3), []).append(w)

    best = None
    for _, ws in sorted(lines.items()):
        ws.sort(key=lambda w: w["x0"])
        text = " ".join(w["text"] for w in ws).lower()
        for rank, label in enumerate(TOTAL_LABELS):
            if label in text and MONEY.match(ws[-1]["text"]):
                # earlier labels are more specific, so they win
                if best is None or rank < best[0]:
                    best = (rank, ws[-1]["text"])
    return best[1] if best else None

# Why this is a ceiling, not a foundation:
#  • "Total" appears on every page footer of some suppliers
#  • "Total excl. VAT" matches "total" and is the wrong number
#  • one supplier prints the amount ABOVE the label, not beside it
#  • credit notes are negative and some print the sign as a suffix
#  • a second currency total in the corner matches just as well

Why matching on labels fails

*Total* is not a unique string. A single invoice can contain *Total excl. VAT*, *Total VAT*, *Total due*, a per-section subtotal called *Total*, and a page footer that repeats *Total* on every page. Matching the first occurrence gives you a number that is confidently wrong; matching the last gives you a different confidently wrong number.

Language multiplies it. The same supplier group sends invoices labelled *Gesamtbetrag*, *Montant total*, *Importe total* and *Totaal*, and some bilingual layouts print two labels beside one figure. A keyword list that covers your current suppliers is a list that needs editing the moment procurement signs a new one.

Position is not a rescue either. The amount is usually to the right of its label, sometimes below it, occasionally in a separate box aligned to a different grid entirely. Every rule you add to handle one supplier is a rule that can misfire on another, and without a per-supplier regression suite you will not know which.

Line items are the hard part

Header fields are a nuisance; line items are a genuine engineering problem. The table has no ruled cells in most modern invoice designs, descriptions wrap across two or three lines, quantity and unit price are right-aligned so wide numbers drift left, and discount or delivery lines appear with a different shape from product lines.

Then the table crosses a page boundary. The header repeats — or does not. A section subtotal appears mid-table and looks exactly like a line. A continuation line carrying only text belongs to the row above, and a naive parser either drops it or promotes it to a row of its own with null amounts, which quietly inflates your line count.

This is where the effort goes if you build it yourself, and it is also where the value is. A register of invoice totals tells you what you spent; a register with lines tells you what you bought, which is what makes cost analysis, project allocation and any kind of spend review possible at all.

flowparse.io

The library choices, and what each costs

The toolkit is the same as for any PDF work in Python, and the trade-offs are practical rather than technical. What decides the choice is usually deployment: a Java runtime or an AGPL dependency is a conversation with someone else, and it is better to have that conversation before the code is written.

The scanned invoice, and the photo of an invoice

If `extract_text()` returns nothing, there is no text layer. For invoices this is common — a supplier prints, signs and scans, or someone photographs a delivery invoice on their phone in a warehouse. The Python route is to render pages and OCR them, or to run OCRmyPDF first so downstream code sees text.

Photographs add problems scans do not have: perspective distortion, uneven lighting, a folded corner across the total. Deskewing and thresholding help, and none of it is free. Every step you add is code that has to be maintained and tuned against documents you have not seen yet.

The honest test is whether your OCR errors are catchable. A misread digit in a total is invisible unless something checks the arithmetic — which is why the validation section is not optional in an OCR pipeline, it is the only thing standing between you and quietly wrong numbers.

flowparse.io

Tax is where invoices get genuinely awkward

One invoice can carry several tax rates, and the correct output is a breakdown, not a number. Standard-rated goods, zero-rated shipping and an exempt line sit happily together, and collapsing them into one `tax_amount` loses information a bookkeeper needs.

Reverse-charge and intra-community supplies print zero tax with a legal note explaining why. Tax-inclusive pricing means the line amounts already contain tax and the net has to be derived. Rounding rules differ — some systems round per line, others on the invoice total — and the difference shows up as a one-cent mismatch that your validation will flag unless the tolerance allows for it.

Do not try to decide deductibility in code. Whether a given tax amount is recoverable is a jurisdictional question about the transaction, not a parsing question, and it belongs with whoever files the return. Extraction's job is to report what the document says, accurately and in full.

The checks that actually catch bad data

Without a running balance, an invoice is checked against itself. Three arithmetic relationships do most of the work: quantity times unit price should equal the line amount, the line amounts should sum to the net, and the net plus tax should equal the gross. Each of them catches a different failure — a misread unit price, a dropped line, a total captured from the wrong box.

Add non-arithmetic checks that cost nothing: a due date that precedes the invoice date means one of the two dates was misread; a missing invoice number on a document that clearly has one means the anchor failed; a total that is negative on something labelled *invoice* rather than *credit note* is worth a second look.

Then deduplicate on supplier plus invoice number rather than on filename. The same invoice arrives twice constantly — attached to a chaser email, forwarded by a colleague, re-sent with a new number by a supplier chasing payment — and the register is where you catch it, before it becomes a duplicate payment.

# An invoice has no running balance, so you check it against ITSELF.
from decimal import Decimal

def check(inv, tol=Decimal("0.02")):
    problems = []
    lines = inv.get("line_items") or []

    for i, l in enumerate(lines):                     # line arithmetic
        q, u, a = (l.get("quantity"), l.get("unit_price"), l.get("amount"))
        if None not in (q, u, a) and abs(Decimal(str(q)) * Decimal(str(u)) - Decimal(str(a))) > tol:
            problems.append(f"line {i + 1}: qty x price != amount")

    if lines and inv.get("subtotal") is not None:     # lines vs net
        s = sum(Decimal(str(l["amount"])) for l in lines if l.get("amount") is not None)
        if abs(s - Decimal(str(inv["subtotal"]))) > tol:
            problems.append(f"line items sum to {s}, subtotal says {inv['subtotal']}")

    if None not in (inv.get("subtotal"), inv.get("tax_amount"), inv.get("total")):
        net, tax, gross = (Decimal(str(inv[k])) for k in ("subtotal", "tax_amount", "total"))
        if abs(net + tax - gross) > tol:              # net + tax vs gross
            problems.append(f"{net} + {tax} != {gross}")

    if inv.get("due_date") and inv.get("invoice_date"):
        if inv["due_date"] < inv["invoice_date"]:     # ISO strings compare correctly
            problems.append("due date precedes invoice date")

    return problems

def dedupe_key(inv):
    """Same supplier + same number = the same invoice, whatever the filename says."""
    supplier = (inv.get("supplier_name") or "").strip().lower()
    number = (inv.get("invoice_number") or "").strip().upper()
    return (supplier, number)

flowparse.io

Should you train a model?

The research answer is yes and the engineering answer is usually no. Layout-aware models — LayoutLM and its descendants, Donut and similar sequence-to-sequence approaches — genuinely solve the layout-variance problem that defeats templates, and open weights are available for all of them.

The cost is the part that is easy to underestimate. Fine-tuning needs labelled invoices, with bounding boxes, in the hundreds; someone has to produce and maintain that dataset. Then you own inference infrastructure, GPU cost or latency, model versioning, and an evaluation loop to prove the new checkpoint is better than the old one on documents you care about.

That is a reasonable investment if document understanding is your product. If invoices are an input to your product, you are building a research capability to solve a purchased problem — and the same money spent on the validation layer would catch more real errors. Either way the arithmetic checks still belong in your code: a model's confidence is not a proof.

When building it yourself is right

Build it when the invoices must stay on your infrastructure. Local processing is a real advantage of the Python route and no hosted API can offer it — if the requirement is absolute, that decides the question before any other argument.

Build it when the supplier set is small and stable. Ten suppliers who each send a consistent layout every month is a bounded problem, and a template per supplier with a regression test per template is entirely maintainable.

Build it when the invoice is machine-generated by a partner you have an agreement with — at which point the better move is to skip PDFs entirely and ask for the data. A CSV, an XML file or an e-invoicing format is not a parsing problem at all, and requesting one is the cheapest fix available.

When to stop building

Stop when the supplier list is open-ended. Every new supplier is a new layout, and a queue that grows faster than you clear it is not a project, it is a tax on the roadmap.

Stop when scans arrive regularly. The OCR pipeline is a separate discipline, and a mediocre one produces confidently wrong numbers rather than honest failures.

Stop when the maintenance is invisible in your metrics but visible in your sprints. Template drift shows up as a slow accumulation of small fixes; it rarely appears in a plan and it never appears in the original estimate.

The same job in fifteen lines

There is no FlowParse Python package, and we do not intend to publish one — the API is a single POST with the document base64-encoded, and a wrapper would be a dependency without a job. What comes back is the normalised invoice: supplier, number, dates, currency, net, tax, total and line items, with dates already ISO and amounts already numbers.

The response also tells you what the document actually is. Send a bank statement to an invoice pipeline and `type` will say so instead of producing plausible nonsense — worth branching on, because in any real inbox a statement, a receipt or a remittance advice eventually turns up in the invoice folder.

The loop below is a working accounts-payable register: extract, reject non-invoices, deduplicate on supplier and number, run the arithmetic checks, and write a spreadsheet with a problems column. That column is the whole point — it is what turns an automated pipeline into one you can supervise.

import base64, os, time, requests
import pandas as pd

API = "https://flowparse.io/api/v1/extract"
KEY = os.environ["FLOWPARSE_API_KEY"]

def extract(path, retries=2):
    with open(path, "rb") as fh:
        payload = {"file": base64.b64encode(fh.read()).decode(),
                   "filename": os.path.basename(path)}
    for attempt in range(retries + 1):
        r = requests.post(API, headers={"Authorization": f"Bearer {KEY}"},
                          json=payload, timeout=180)
        if r.status_code == 200:
            body = r.json()                 # {type, pages, billedPages, data:{type,data}}
            return body["type"], body["data"]["data"]
        if r.status_code in (400, 422):     # bad request / nothing extractable (not billed)
            raise ValueError(r.json().get("error", "unreadable document"))
        if r.status_code == 429:            # out of pages — retrying cannot help
            raise RuntimeError(r.json().get("error", "page budget exhausted"))
        if r.status_code >= 500 and attempt < retries:
            time.sleep(2 ** attempt)
            continue
        r.raise_for_status()

register, seen = [], set()
for name in sorted(os.listdir("inbox")):
    kind, inv = extract(os.path.join("inbox", name))
    if kind != "invoice":                   # a statement or a receipt wandered in
        print(f"{name}: got {kind}, routing elsewhere"); continue
    key = dedupe_key(inv)
    if key in seen:
        print(f"{name}: duplicate of {key[1]}"); continue
    seen.add(key)
    register.append({
        "file": name,
        "supplier": inv.get("supplier_name"),
        "number": inv.get("invoice_number"),
        "date": inv.get("invoice_date"),
        "due": inv.get("due_date"),
        "currency": inv.get("currency"),
        "net": inv.get("subtotal"),
        "tax": inv.get("tax_amount"),
        "total": inv.get("total"),
        "lines": len(inv.get("line_items") or []),
        "problems": "; ".join(check(inv)) or "",
    })

pd.DataFrame(register).to_excel("invoice-register.xlsx", index=False)

Status codes, retries and what each call costs

Three codes need deliberate handling. A 422 means nothing extractable was found — a blank scan, a photograph of a wall, a PDF that is really a letter — and it is not billed. A 429 means the page budget is exhausted, and no amount of retrying produces pages. A 400 usually means the payload is malformed, most often a base64 problem.

Only 5xx deserves a backoff. This matters more than usual because every retry that reaches the API is a real extraction against your allowance: a naive `for attempt in range(5)` around every call turns one transient blip into five billed reads of the same document.

Nothing is deduplicated on our side either. Sending the same invoice twice extracts and bills twice, because the API has no way to know you did not mean it. Keep the `seen` set — or a small table — in your own pipeline, and key it on supplier plus invoice number rather than on the filename, which changes every time somebody forwards the email.

Size, time and the shape of a good request

Documents are capped at 20 MB, which ordinary invoices never approach and a scanned bundle of forty pages does. Extraction has an internal limit of 110 seconds, so send invoices individually rather than concatenated — a merged PDF of a month's invoices is both slower and harder to attribute when one page fails.

Set the client timeout above the server's. A short `requests` timeout on a legitimately slow document abandons the connection while extraction completes and bills — the worst possible outcome. Something around 180 seconds, with splitting for anything genuinely large, is the sane configuration.

If you are calling from an automation platform rather than Python, its window is usually tighter than ours: Power Automate stops at two minutes and Airtable at thirty seconds. Those pages cover the same call under those constraints.

What to do with the output

The most useful artefact is an invoice register: one row per document with supplier, number, dates, currency, net, tax, total, line count and any validation problems. It is the thing an accountant asks for, it is what makes a month reviewable, and it takes one `to_excel` call once the extraction is done.

Keep the line items too, in a second sheet or table keyed by invoice number. Extracting again later costs another page, and the question that needs the lines always arrives after the month is closed.

From the register, everything else is downstream: a payables ageing, a spend-by-supplier summary, an import file for a ledger. The invoice PDF to Excel page covers the no-code version of the same output, and how to create an invoice register from PDFs covers the process rather than the code.

flowparse.io

Testing an invoice parser

Commit real invoices as fixtures, one per supplier layout, anonymised if you must but never synthetic — generated PDFs do not reproduce the quirks that break parsers. For each, assert the header fields, the line count and the three arithmetic relationships. That is a small file per fixture and it catches nearly everything.

Test the failures deliberately as well: a credit note, an invoice with two tax rates, one with a table that crosses a page, one scan. Those four documents represent the majority of production surprises, and a suite without them proves only that clean invoices work.

Track a production metric alongside the tests — the share of documents that pass all arithmetic checks, by supplier. A supplier whose pass rate drops has changed something, and that is a signal no unit test will give you.

The arithmetic, with your own numbers

The comparison is not about capability, because both routes can be made to work. It is about where your engineering time is worth more, and about which failures you are willing to own.

The boundary, stated plainly

No Python SDK, no PyPI package, no plans for either. No self-hosted, on-premise or air-gapped deployment, and the engine is not open source — if invoices may never leave your infrastructure, the libraries in this page are your route and we are not a candidate.

No approval workflow, no purchase-order matching, no supplier master and no payment execution. The API returns data; deciding whether an invoice is correct, approved and payable belongs to your process. Three-way match and AP automation describe where that boundary sits in a wider process.

And no archive: originals are deleted immediately after extraction. The supplier's PDF is the evidence for tax and audit purposes, the extracted data is a working dataset, and keeping the former is your obligation, not ours.

A worked example: a marketplace with 400 suppliers

A procurement platform needed invoice data from suppliers onboarded by its customers — an open-ended list, growing weekly, with no control over what software any of them used. They started with templates because the first fifteen suppliers were large and consistent, and they built a decent internal tool.

It broke at scale in a specific way. Coverage was fine for the top suppliers and terrible in the long tail, so the numbers looked healthy in aggregate while the customers with many small suppliers had a poor experience. Adding templates for the tail was never worth an individual engineer's week, so the tail never got better.

What they kept from the build: the validation layer, the deduplication key, the register format and their fixture suite, all of which are the parts that encode their domain knowledge. What they replaced was the extraction. The arithmetic checks now run against API output, and the metric they watch is the share of invoices passing all three checks per supplier — which is a better product metric than parser coverage ever was.

Where to go next

The full response schema and every endpoint are in the API docs; keys come from the API dashboard. The invoice parsing API page covers invoice-specific behaviour and the document extraction API covers the mixed-inbox case.

The statement counterpart of this page — different failure modes, and a completeness proof this one does not have — is bank statement parser in Python. For the rule set behind the checks, see invoice validation.

If you would rather not write any of it, invoice PDF to Excel does this by upload, batch does a folder at a time, and extract invoice data is the field-by-field overview.

Frequently asked questions

Related