AI Agent for Document Classification

A scanned AP inbox mixes invoices, purchase orders, contracts, and receipts with no consistent structure — different vendors, different layouts, different scan quality. The job of a document classification agent is narrow but high-stakes: look at an incoming file, decide what it is and where it belongs in the ERP workflow, and know when it isn't sure enough to guess. Get the confidence handling wrong and you either bury AP clerks in false positives or silently misroute a real invoice.

The pipeline: OCR, then classification

Classification starts before any LLM call. A scanned or emailed PDF goes through OCR (Azure Document Intelligence, AWS Textract, or an open-source engine like Tesseract for simpler layouts) to produce both raw text and, ideally, layout metadata — bounding boxes, table structure, key-value pairs. The LLM classification step then works off the OCR text plus a handful of structural signals, not the raw image. This matters because OCR quality is the real bottleneck: a blurry fax-quality scan degrades every downstream decision, and no amount of prompt engineering fixes text that OCR got wrong.

For document type classification specifically, you don't need a general-purpose LLM to do the heavy lifting. A cheap first pass — regex on known vendor letterhead, PO number patterns, keyword presence ("Remit To", "Bill of Lading", "Net 30") — resolves a large share of documents deterministically. Reserve the LLM call for the ambiguous remainder, where it earns its cost.

Classification with confidence scores

The classification call itself uses structured output (tool calling / JSON schema) rather than free text, so the result is directly consumable by the workflow engine. Ask the model to return a document type, a confidence score, and — critically — the specific evidence it used, so a reviewer isn't starting from zero when they check the agent's work.

classify_document_tool = {
    "name": "classify_document",
    "description": "Classify an OCR'd AP document into a known type.",
    "input_schema": {
        "type": "object",
        "properties": {
            "document_type": {
                "type": "string",
                "enum": ["invoice", "purchase_order", "contract", "receipt", "unknown"]
            },
            "confidence": {"type": "number", "minimum": 0, "maximum": 1},
            "vendor_name": {"type": "string"},
            "evidence": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Short quotes from the OCR text that support the classification"
            }
        },
        "required": ["document_type", "confidence", "evidence"]
    }
}

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=512,
    tools=[classify_document_tool],
    tool_choice={"type": "tool", "name": "classify_document"},
    messages=[{"role": "user", "content": f"OCR text:\n{ocr_text}"}]
)

The confidence field is not the model's raw token probability — LLMs are notoriously overconfident when asked to self-report — so treat it as a soft signal and calibrate it against a labeled validation set. Run a batch of a few hundred historical documents with known ground truth, plot predicted confidence against actual accuracy, and pick your routing threshold off that curve rather than a round number that feels right.

Routing by confidence threshold

Once a document is classified, it routes one of three ways: high confidence goes straight into the matching workflow (e.g., an invoice enters 3-way match against PO and receipt automatically); mid confidence goes into a human review queue with the agent's classification pre-filled as a suggestion; low confidence or "unknown" goes into a manual triage queue with no suggestion at all, because a wrong suggestion anchors the reviewer more than no suggestion does.

The human review queue

The review queue is not a dumping ground — its design determines whether the system actually saves time. Each item needs the original document image side by side with the extracted fields, the agent's stated evidence, and a one-click accept/correct action. When a reviewer corrects a classification, that correction is the most valuable training signal you have: log it, and periodically review the correction log for a vendor or document type that's consistently misclassified, because that's usually a sign the OCR layout for that vendor needs a specific handling rule, not a smarter prompt.

Guardrails and audit trail

Every classification decision — auto-routed or human-reviewed — needs to be logged with the document ID, the model version, the confidence score, and the reviewer if one was involved. This isn't optional bookkeeping; when a vendor disputes a payment or an auditor asks why an invoice was processed without a PO match, you need to reconstruct exactly what happened and why the system trusted it. Version the classification prompt and schema alongside the code, because a prompt change that shifts confidence calibration is functionally a model change and should be tracked the same way.

Wrapping up

Document classification for AP is a good candidate for automation precisely because the failure mode is cheap to catch — a misrouted document sits in a queue, it doesn't silently pay the wrong vendor. The work that actually matters is calibrating thresholds against real data, designing a review queue that makes correction fast, and keeping an audit trail thorough enough to answer "why did the system do that" six months later.