Building Data Extraction Pipelines for Legacy Documents

A practical guide to combining Python, OCR, visual evidence and domain knowledge to turn archived documents into structured records.

A legacy PDF can contain readable text, tables and screenshots without making their relationships clear. A date beside the final total may disagree with the period on the detail rows. An embedded table may lose a column heading at its edge. OCR can recognize the characters correctly while the extraction assigns them the wrong meaning.

We encountered these problems while migrating archived financial statements from WebCRM into PostgreSQL for commission reporting. The difficult part was producing a complete reading of each report and deciding how its evidence should govern the resulting records. Historical errors and missing information had to remain visible throughout that process.

This guide explains the method we developed: prepare complementary views of each document, turn domain knowledge into extraction instructions, and review the source representation before mapping it into the destination. The migration provides the running example. The code examples are abbreviated, with pseudocode labeled where application-specific helpers stand in for the full implementation.

See What OCR Misses

Start With the Difficult Documents

Begin with documents that expose the archive's different reading problems. A clean text report tells you little about a statement assembled from small screenshots. Look for mixed layouts, dense tables, conflicting headings and content close to page boundaries. Inspect these examples before committing to an OCR configuration or an extraction prompt.

We checked four representative PDFs with Mistral OCR, pinned to mistral-ocr-4-1. We then sent the remaining 1,920 PDFs from the broader archive through sixteen asynchronous batches using Mistral's Batch API. Identical files were processed once, with their occurrences in the archive kept linked. We saved the responses so later extraction runs could reuse the OCR work.

This prepared the broader archive. We selected a smaller subset of financial documents for extraction and import.

Keep the Table Intact

The OCR configuration requested HTML tables, layout blocks, headers, footers and embedded images. We converted the tables into fixed rows and columns and gave each block a stable reference to its page. Blank cells and column positions survived alongside the text.

That structure carries meaning. A code placed under the neighboring column represents a different field even when every digit is correct. Flattening a table into prose removes some of the evidence needed to distinguish those readings. Retain the row, column, heading and source location so a later interpretation can be checked against the report.

Keep the original OCR response as well as the representation prepared for the model. The response records what the OCR service returned. The prepared representation supplies a consistent way to refer to its content across instructions, model output and review.

Cross-Check the Visible Page

We gave the extraction model complementary views of each PDF. Alongside OCR, Poppler's pdftotext -layout supplied the PDF's native text with approximate spacing and page boundaries. This provided another reading of signs, identifiers and lines that OCR might omit. Embedded screenshots have no native text, so they still needed OCR and visual inspection.

Poppler's pdftoppm rendered full pages at a maximum dimension of 2,400 pixels. Every target page took priority over supplementary views and teaching-example images. The remaining image budget went first to close-ups of embedded image regions, then to regions where native text and OCR disagreed. These views were rendered at 300 DPI. A close-up helped read a small region, while the full page established where it belonged.

The close-ups were rendered through the PDF's visible layout. This is an important distinction: underlying text objects and image resources can contain material that the page clips away. Native text can help resolve a character, but the rendered page establishes whether that character is visible in the report.

The following pseudocode shows the preparation order. The OCR input is a saved response, and the helpers represent the application's document-processing code.

def prepare_evidence(pdf, saved_ocr, image_budget):
    blocks = structure_with_source_refs(saved_ocr)
    native_text = read_native_text(pdf, layout=True)
    pages = render_all_pages(pdf, max_dimension=2400)

    if len(pages) > image_budget:
        raise NeedsPageGrouping(pdf)

    images = list(pages)
    closeups = render_visible_regions(pdf, dpi=300)
    images += closeups[:image_budget - len(images)]

    return blocks, native_text, images

The guard makes the priority explicit: a request should not silently lose target pages to make room for supplementary images. Documents that exceed the chosen budget need a separate grouping strategy that preserves the relationships between pages. In our harness, teaching-example images used the capacity left after target pages and close-ups. Their OCR, reviewed output and decision notes remained in the request even when their images did not fit.

Make Domain Judgment Reusable

Decide Which Evidence Governs the Record

Several views of a document help the model read it. They do not, by themselves, establish which period governs a commission record or whether an unlabeled number represents a count. Those decisions require knowledge of the surrounding system.

I worked with GPT-6 Astra in Codex to investigate the sources and build the extraction method. My familiarity with the application helped me recognize when a plausible reading missed a financial convention. Astra could trace the implementation, compare representations and test an explanation against the documents. The useful exchange was making each uncertainty specific enough to investigate together.

For example, a statement could contain a period on its detailed insurance rows, another period in an embedded report and a different caption beside its final total. I knew that the period on the property and casualty insurance details had authority for this commission history. Choosing it connected the statement to the salesperson's historical record and the rates for that month.

We made the rule explicit. Extraction retained the printed labels and their locations. Mapping later used the reviewed authoritative period to determine the destination month. Conflicting labels remained available for review, and exceptional cases required a recorded decision tied to the source.

For another archive, establish this authority from its own business process. A date's position, size or proximity to a total alone does not establish what it governs. Work through an actual conflict and record the evidence that justifies the choice.

Preserve Errors and Unknowns

The same principle applies to amounts. A historical statement's detail figures can disagree with its printed payout. Our extraction policy retained both, including their original signs and precision. Correcting the arithmetic during extraction would have replaced the report's evidence with a new calculation.

Missing meaning also needs an explicit representation. A pension report in our archive contained readable numbers without headings that established their roles. A value that resembled a member count remained an unknown when the source did not identify it. We retained the value and its location so later review could revisit the question.

This becomes especially important when the destination requires a field the source does not supply. A database requirement cannot make a historical fact appear. Keep the source extraction incomplete where the evidence is incomplete. Any later derivation or estimate belongs in the mapping policy, with its assumptions recorded separately.

The reading instructions also required comparison of the same physical occurrence across the page image, available close-up, native text and OCR block. The model had to check both directions: OCR content without visible support, and visible content that OCR had omitted. This covered small rate labels, blank columns, repeated headings and fragments at screenshot edges. Differences had to be resolved from the source, with remaining uncertainty retained.

Turn Decisions Into Worked Examples

Once a case is understood, make the decision reusable. We assembled three reviewed source examples, each pairing document evidence with the expected structured output and notes about the interpretation. The notes identified the evidence, the applicable rule and the resulting extraction decision.

A condensed lesson for the period conflict looks like this. This is an illustrative decision note, not a raw record from the archive.

evidence:
  - Period on property and casualty detail rows
  - Conflicting caption beside the final total
rule:
  - Retain both labels with their source locations
  - Detail-period evidence governs destination mapping
reason:
  - >-
    The period selects the applicable
    commission history and rates
extraction:
  - Identify each label and its role
  - Preserve the conflict for review
mapping:
  - Use the reviewed authoritative period

The example gives a subsequent call more than a desired answer. It explains which evidence matters, why it matters and what must survive in the output. A reviewer can inspect the same evidence and challenge the rule or its application.

We also supplied a small classification reference from the application: five stored rows for each of five insurance categories. These showed how the application's vocabulary related to real records. The model could use them to interpret a field while taking its value from the current document.

This is where the human-and-agent collaboration produced something useful beyond the investigation. We selected decisions from the work, checked them against the sources and turned them into focused teaching material. The full session history was a record to revisit. The extraction calls received curated instructions and examples as context. We did not train a new model.

From One PDF to the Whole Archive

Give Each Extraction One Clear Job

We built the document-reading harness with PydanticAI, using Kimi K3 at maximum reasoning effort through Baseten's Model APIs. The harness assembled each document's evidence with the reading instructions, three worked examples and classification references. It limited model requests and saved responses for review.

The output represented the document as pages, blocks, rows, cells and fields. Roles distinguished details, subtotals, adjustments and period labels. A corrected OCR reading retained the original text, its replacement and its source location. This structure let review follow an interpretation back to the document that supported it.

The extraction model had no tools. It read and classified the source, while Python applied the reviewed mapping rules and calculations. This kept interpretation of the document separate from decisions about the destination records. A disagreement in the report could survive extraction and be handled explicitly during mapping.

Our PydanticAI configuration used PromptedOutput to describe the output schema in the prompt and parse the returned JSON locally. The abbreviated code below follows the archived harness. Each invocation creates a fresh agent for one PDF. The model connection, output schema, reading instructions and that document's evidence prompt are prepared separately.

from pydantic_ai import Agent, PromptedOutput
from pydantic_ai.usage import UsageLimits

async def extract_one_pdf(model, evidence_prompt):
    extractor = Agent(
        model,
        output_type=PromptedOutput(ExtractedPDF),
        instructions=reading_contract,
        retries={"tools": 0, "output": 2},
    )

    return await extractor.run(
        evidence_prompt,
        usage_limits=UsageLimits(
            request_limit=3,
            tool_calls_limit=0,
        ),
    )

Separate “the output has the right structure” from “the underlying claims are correct.”

Passing the output schema does not establish that an amount belongs to the stated period, or that every visible row was included. Those questions require comparison with the source. Source references in the output let a reviewer trace each interpretation back to its evidence.

Anthropic's guide to agent evaluations distinguishes the harness that runs an agent from the evaluation harness that tests it. The object of evaluation is the model and its harness working together. For document extraction, this means checking the reading produced with the supplied page images, OCR, instructions and examples. Changes to evidence preparation, instructions or examples need to be evaluated with the model that will use them.

How to Scale the Task

Our unit of work was one complete PDF per agent instance. Each attempt had its own document context, prompt and output, with no conversation history from another target document. The shared instructions and reviewed examples gave every instance the same method to apply.

Work size determines how much evidence the model must connect at once. A complete PDF kept headings, detail rows and totals together across pages. Adding unrelated reports would introduce dates and figures that the current extraction did not need. Splitting every page into a separate job could remove the context that explained a table. For this archive, PDFs ranging from one to eight pages gave us a useful boundary.

Five controls shaped how we scaled that work:

1. Work per instance. Choose the smallest unit that contains the evidence needed for a complete result. Here, each instance read one whole report. Longer document families may need sections grouped around tables, headings and their supporting pages.
2. Evidence budget. Our frozen configuration allowed eight images per request, with target pages reserved first. This was our chosen budget, not the model's capacity limit. Close-ups used the remaining slots to make dense regions readable. Increasing throughput should preserve that evidence within each job.
3. Reasoning and response space. We used maximum reasoning effort and a 64,000-token allowance per request, including reasoning. The response needed room to represent the entire document. Choose these settings by checking difficult examples for complete, source-faithful readings. A larger allowance alone does not establish a better result.
4. Concurrent jobs. Up to six independent document jobs ran at once. A shared scheduler paced their requests within the account's token and request limits. This let us process more PDFs while keeping each instance's assignment fixed. The number of active jobs and the number of requests admitted to the provider were separate controls.
5. Review and retry scope. Each document retained its own output, source references and attempt history. Review or another attempt could address that document independently, while completed outputs stayed available for the rest of the archive. One PDF could require several model requests, so one instance did not imply one call.

The aim is to increase the number of complete, focused readings the system can produce at once. Establish a coherent document boundary and verify the result first, then raise concurrency within the provider's capacity. Work size and parallelism become quality levers when they let each instance retain the evidence and response space it needs as the archive grows.

Inspect and Reveal Visually Hidden Information

Using several representations lets us inspect the completeness of each one. A page image preserves the layout, native text offers another reading of its characters, and OCR makes text inside screenshots available for structured extraction. Comparing the same region across these views can reveal a missed label, clarify a sign or show where content disappears at a table's edge.

Extracting embedded images takes that inspection beyond the displayed page. In our source review, some underlying bitmaps retained readable rows that the PDF's layout cropped away. Others were already missing the relevant content. Comparing the extracted image with the rendered page established whether the information could be recovered from the file or was absent from that embedded source.

Verify each recovered reading against its surrounding rows and headings, then record which representation exposed it and where it belongs. We retained content found beyond the visible crop as separately reviewed evidence, alongside the model's original extraction. This preserved the distinction between what the report displayed and what additional inspection established, giving the database mapping an explicit basis for using that information.

Carry the Evidence Into the Database

With the source representation reviewed, mapping can apply the destination's conventions. In our migration, it assigned the authoritative month and connected each statement to the appropriate historical record. Missing destination values could be derived under approved rules, including estimates based on later system behavior. Those remained labeled reconstructions.

Where extracted components disagreed with the printed payout, we used explicit balancing rows to represent the difference. The source details stayed intact. The destination could reconcile to the historical payout while preserving the disagreement that made the balance necessary. Where individual details were unavailable, separately identified aggregate rows represented the accepted source evidence.

These choices have consequences for downstream use. A balancing row can contribute to a row-count metric even though it represents no observed sale. Decide how the destination should treat such records and retain their kind and provenance. Reconciliation to a printed total establishes agreement with that control, while the accuracy of the underlying details remains a separate question.

The resulting extraction package covered 386 financial PDFs. Combining these with three spreadsheets and selecting the authoritative revisions produced 383 statements covering January 2024 through February 2025.

The final import contained 20,294 destination rows, including 431 balancing rows and 41 aggregate rows. All 381 available printed final totals reconciled with those representations included. The remaining two statements had component totals but no printed final total. A separate read-only check confirmed the committed production records.

Human review remained part of the completed backfill. We did not measure total time savings or isolate how task size, model settings and teaching notes affected accuracy.

What Comes Next

The work showed how domain judgment can become reusable when decisions stay tied to their evidence. Working through the documents with Astra helped us turn choices about source authority, historical values and uncertainty into instructions and reviewed examples. Subsequent model calls received both the evidence to read and the rules for interpreting it.

Next, I want to test the method on unfamiliar document families, comparing the same model and evidence with and without the decision notes attached to the examples. Measuring extraction accuracy and the human review required would show whether those explanations improve results beyond the original archive. That would give us a basis for applying the approach to other tasks where expertise is easier to demonstrate in a real case than to describe in advance.