Automate document extraction with webhooks

First, which way do the webhooks point?

This trips up almost everyone who searches for this topic, so it is worth being blunt before you write any code.

That is a deliberate design rather than a missing feature, and it makes your life easier in one important way: there is no delivery to verify, no callback signature to check, no partial state where we think we told you and you never heard. The request either returned data or it did not, and your code knows immediately which.

What people actually mean when they search for webhook-driven document extraction is the inbound direction: a document arrives at a URL they control, and that arrival triggers the pipeline. That is a real and extremely common pattern, and it is what the rest of this guide builds. The webhook starts the work; the extraction call does the work.

The architecture in one picture

Five parts, and the second one is the part people skip and later regret. A webhook handler that does the extraction inline will eventually time out — extraction takes seconds, and most webhook senders expect an answer in well under one. When it times out, the sender retries, and now you have the same document being processed twice. The queue is not architectural decoration; it is what makes the design correct.

flowparse.io

Before you start

You need an API key, which is free to create at get an API key, and a page balance for the extraction calls themselves. Validation, reconciliation and export previews cost nothing, so most of what follows can be built and tested for free — a point worth using deliberately rather than discovering afterwards.

On your own side you need somewhere to run a webhook endpoint and a worker, plus a queue. The queue does not have to be elaborate: a database table with a status column, polled by a worker, is entirely sufficient and easier to reason about than a message broker you have not run before. What matters is that a job is durable, claimable and retryable — not which technology provides those properties.

If you would rather not run any of that, skip to the no-code section — Make and n8n both provide the webhook, the queue and the worker as parts of a scenario you draw rather than deploy.

1. Receive the document

Your webhook handler has exactly three jobs, and doing more than these three is the most common design error in this pipeline. Verify that the request is genuine. Persist the document somewhere durable. Enqueue a job and return. It should not call the extraction API, it should not write to your ledger, and it should not send an email.

Verification matters because a webhook URL is, by definition, a public endpoint that accepts data. Use a shared secret and an HMAC signature computed over the raw body — not the parsed JSON, since re-serialising changes the bytes — and compare it with a constant-time function so you are not leaking information through timing. Reject anything whose timestamp sits far outside a tolerance window, which is what stops a captured request being replayed later.

import crypto from "node:crypto"

export async function POST(req: Request) {
  const raw = await req.text()                       // raw bytes, not parsed JSON
  const sig = req.headers.get("x-signature") ?? ""
  const expected = crypto.createHmac("sha256", process.env.WEBHOOK_SECRET!)
    .update(raw).digest("hex")

  // Constant-time compare — never ===
  const ok = sig.length === expected.length &&
    crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected))
  if (!ok) return new Response("bad signature", { status: 401 })

  const { fileBase64, filename } = JSON.parse(raw)
  const hash = crypto.createHash("sha256").update(fileBase64).digest("hex")

  // Idempotency: the same document twice is the same job, not two jobs.
  const job = await db.job.upsert({
    where:  { hash },
    create: { hash, filename, fileBase64, status: "PENDING" },
    update: {},                                      // already seen — do nothing
  })

  return Response.json({ jobId: job.id }, { status: 202 })
}

2. Queue the work

The queue exists to absorb the difference between how documents arrive and how fast they can be processed. Documents arrive in bursts — a month-end batch, a client uploading a year of statements in one sitting — and extraction takes seconds per document. Without a queue, that mismatch becomes timeouts; with one, it becomes a slightly longer wait.

A database-backed queue is enough for almost everyone, and it has a property message brokers make harder: you can look at it. When somebody asks why a document has not appeared, a row with a status, an attempt count and a last error answers the question in one query. Keep the statuses few and meaningful.

One detail worth building in from the start: a claimed job that never completes — because the worker was restarted mid-flight — must return to PENDING after a timeout, or it sits in PROCESSING forever. A reaper that resets stale claims is twenty lines and saves a genuinely confusing class of support ticket.

flowparse.io

3. Extract

The worker claims a job, base64-encodes the document and POSTs it. This is the only call in the pipeline that costs money, and it is the only one that needs the file itself. Classification is automatic, so you do not have to know in advance whether you are holding an invoice, a receipt or a bank statement — the response tells you in type.

const res = await fetch("https://flowparse.io/api/v1/extract", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.FLOWPARSE_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ file: job.fileBase64, filename: job.filename }),
})

if (res.status === 422) return markFailed(job, "unreadable")      // not billed
if (res.status === 429) return pauseQueue("page budget exhausted")
if (!res.ok)            return scheduleRetry(job, res.status)

const { type, pages, billedPages, data } = await res.json()

Note what the worker does not do: it does not retry a 422, because a document that cannot be read will not become readable on the second attempt, and it does not retry a 429, because your budget will not refill by trying again. Those two distinctions are what separate a worker that recovers from one that spins.

For document-specific behaviour, the invoice parsing API, receipt OCR API and bank statement API each document their own schema; all three are the same endpoint with a different type in the response.

flowparse.io

4. Validate — the step that makes it safe

An unattended pipeline without a quality gate is a fast, reliable way to get wrong numbers into your books. Nobody downstream questions a figure that arrived automatically; an extraction that dropped four transactions looks exactly like one that did not. So before anything is delivered, score it.

/api/v1/validateis free and deterministic — arithmetic and rules rather than a second model opinion, so the same data always produces the same verdict. For bank statements it checks the extracted transactions against the statement's own opening and closing balance, which is the only check that can prove an extraction incomplete with no labels and no human comparing anything. For invoices it checks line amounts against the subtotal and subtotal plus tax against the total.

const v = await fetch("https://flowparse.io/api/v1/validate", {
  method: "POST",
  headers: { "Authorization": `Bearer ${process.env.FLOWPARSE_KEY}`,
             "Content-Type": "application/json" },
  body: JSON.stringify({ type, data }),
}).then(r => r.json())

const score = v.validations?.[0]?.score            // { value: 96, grade: "A" }

if (score.grade === "A" || score.grade === "B") {
  await deliver(job, data, score)                  // straight through
} else {
  await markNeedsReview(job, data, score.checks)   // a person, with the failing check named
}

flowparse.io

5. Deliver

Delivery is where the pipeline meets whatever you actually needed. For most builds that is a database write — transactions into a table, invoice headers and line items into two. For others the destination is a file: call /api/v1/export with a format such as xlsx, qbo or xero and you get the finished file base64-encoded, ready to save or email. The PDF to Excel API covers that path in detail.

Whatever the destination, store the raw extraction JSON alongside it. It costs almost nothing and it buys two things you will want: the ability to answer "where did this number come from" with evidence rather than reconstruction, and the ability to regenerate an export in a different format later without paying to read the document again.

This is also the point where your own outbound webhook makes sense, if you have downstream systems that care. Your pipeline knows a document is finished; nothing else does until you tell it. That notification is yours to design, and unlike the inbound one it can be as slow as you like.

flowparse.io

Idempotency: the one thing you cannot retrofit

Everything in a distributed pipeline gets delivered twice eventually. A sender times out waiting for your 202 and retries. A worker crashes after calling the API but before writing the result. Somebody forwards the same invoice email a second time. If your pipeline treats each arrival as new work, every one of those becomes a duplicate record — and a duplicate bill in an accounting system is a genuinely expensive mistake.

The fix is a hash of the file contents used as a natural key. Compute it at the webhook, store it on the job, and make the insert an upsert. The same document arriving twice then produces the same job, and a retry returns the stored result instead of extracting — which also means you are not billed twice for the same pages.

Hash the file content rather than a filename or a message id. Filenames repeat, message ids differ between the two deliveries of the same document, and only the bytes tell you it is genuinely the same file. Deduplicating within your ledger as well is worth doing — see duplicate payment detection for the accounting-side control.

Retries and backoff

Retry the failures that might succeed next time, and only those. A transient 503 or a dropped connection deserves another attempt after a pause. A 400 will fail identically forever, because the request itself is wrong. A 422 means the document is unreadable, which will still be true in thirty seconds. A 429 means your budget is empty, and retrying cannot refill it.

Use exponential backoff with jitter for the retryable cases. Jitter matters more than people expect: without it, a burst of jobs that all failed at the same moment will all retry at the same moment, reproducing exactly the spike that caused the failure. Cap the attempts and move the job to FAILED with the last error recorded, so an unfixable document ends up inspectable rather than looping forever.

Error handling that keeps documents

The measure of a document pipeline is not how it behaves when everything works — it is whether a document can ever disappear. Every failure path should end in a row somebody can find: NEEDS_REVIEW for extractions that failed the quality gate, FAILED for documents that could not be read at all, each with the reason attached.

Treat FAILED as a verdict about the document rather than an error in your system. "This scan is too poor to read" is useful, actionable information; a job stuck in PROCESSING because a worker died is not, which is why the reaper matters. Distinguishing the two in your status model means your alerting can ignore the former and page on the latter.

Alert on rates, not events. One unreadable receipt is normal. A 422 rate that has tripled since Tuesday means something changed upstream — a client switched to photographing statements, a scanner is set to a lower resolution — and catching that in days rather than at month end is the whole value of monitoring it.

flowparse.io

Building it without code

Every part of this architecture exists as a node on the automation platforms, which means you can have the same design without deploying anything. The webhook trigger receives the document, the platform's own execution model provides the queue and retries, an HTTP node calls the extraction API, and a router or IF node is your quality gate.

The trade-off is the usual one. A platform removes the operational burden and costs per task or per operation, which adds up at volume; code costs engineering time and then runs for the price of the server. A useful rule of thumb: if the document volume is measured in hundreds per month, use a platform; if it is thousands per day, the per-task meter will make the case for code on its own.

Worked example: supplier invoices end to end

A finance team receives supplier invoices at a shared address. They want them in their ledger the same day, with anything doubtful reviewed rather than posted. Here is the whole pipeline as it would actually be built.

A mailbox integration POSTs each attachment to their webhook. The handler verifies the signature, hashes the file, upserts a job and returns 202 — total time a few milliseconds, so the sender never times out. A worker polls for PENDING jobs, claims one, and calls extract. The response comes back classified as an invoice with supplier, number, dates, totals and line items typed and ready.

The worker then calls validate. Most invoices score an A: the line amounts sum to the subtotal, subtotal plus tax equals the total, nothing is missing. Those are written to the ledger staging table and marked DONE. A handful score lower — usually a supplier whose scan is poor, or an invoice with a handwritten adjustment — and those become NEEDS_REVIEW with the failing check named, so the reviewer looks at one flagged number rather than re-reading the document.

What makes this defensible rather than merely automatic is the audit trail. Every row carries the validation grade, the billed pages and the raw extraction. When an auditor asks how a number reached the books, the answer is a query rather than a reconstruction — and the exceptions were never silently swallowed. The same shape works for statements, where the balance check is even stronger evidence; see parsing bank statements with an API.

flowparse.io

Scaling it up

Scaling this design is mostly about adding workers, because the queue already decouples arrival from processing. Documents are independent, so extraction parallelises cleanly — send each one as its own request rather than assembling a giant batch, and one corrupt file cannot cost you the rest.

The constraint that bites first is not throughput, it is budget. Twenty workers will consume pages twenty times faster, which is excellent until something loops. Cap the pool size deliberately, and consider a simple spend guard: if the pages consumed in the last hour exceed what you expect in a day, pause and alert rather than continue. That single check has saved more money than any optimisation.

When the output should be one file rather than many, /api/v1/merge consolidates up to 100 extracted documents into a single Excel workbook — the API side of Smart Merge. Previews are free, so the consolidated layout can be checked before anything is billed.

flowparse.io

Security

Three things, in order of how often they go wrong. First, the webhook: verify signatures over the raw body with a constant-time comparison, and reject stale timestamps. An unverified webhook endpoint is an open door into your pipeline.

Second, logs. This is the failure we see most often, and it is almost always accidental: a debug line that logs the request body, left in place, quietly writing bank statements into a log aggregator that far more people can read than can read your database. Log identifiers, statuses and durations. Never bodies.

Third, keys. Use a separate API key per environment and per integration, so a leaked staging key is revoked without touching production and per-key usage tells you which system is consuming what. On our side requests travel over HTTPS, keys are stored hashed, uploaded files are processed to produce the response rather than retained as downloadable documents, and nothing is used to train models — the security page has the detail.

Common mistakes

These are the ones that actually appear in production, roughly in order of frequency.

Best practices

Keep the webhook handler trivial and the worker interesting. Hash file contents for idempotency before you need it. Validate everything and branch on the result — the call is free and it is the only thing standing between an automated pipeline and automated mistakes. Store the raw extraction next to whatever you derive from it.

Operationally: cap concurrency, back off with jitter, distinguish retryable from terminal failures, reap stale claims, and alert on rates rather than events. Build the entire pipeline against free validation and preview calls first, then switch on billed extraction when it behaves — an approach that costs nothing and finds most design errors before they cost anything either.

And keep a human in the loop where it earns its keep. The point of the quality gate is not to eliminate review but to spend it only where it changes an outcome — a reviewer looking at eight flagged documents out of four hundred is a good use of attention; the same person checking all four hundred is the manual process you were trying to replace.

flowparse.io

Frequently asked questions

Keep reading