AI Agent for Email Triage

A shared support, sales, or AP inbox gets dozens to hundreds of emails a day, and most of the value in triaging them isn't answering — it's routing correctly and extracting the right facts fast. An email triage agent reads each message, classifies it, pulls out structured entities, and decides whether to draft a reply, auto-route it, or hand it to a person untouched. The hard part isn't the classification; it's making sure the agent never sends something wrong on its own authority.

Classification and entity extraction

The first LLM call does two things in one pass: assigns a category (support, sales lead, AP invoice query, spam, other) and extracts entities relevant to that category — a ticket or order number, an invoice reference, a dollar amount, a sentiment signal. Doing both in a single structured call is cheaper than separate passes and keeps the extraction grounded in the same classification reasoning.

triage_tool = {
    "name": "triage_email",
    "input_schema": {
        "type": "object",
        "properties": {
            "category": {
                "type": "string",
                "enum": ["support", "sales_lead", "ap_invoice_query", "spam", "other"]
            },
            "urgency": {"type": "string", "enum": ["low", "normal", "high"]},
            "entities": {
                "type": "object",
                "properties": {
                    "order_number": {"type": ["string", "null"]},
                    "invoice_number": {"type": ["string", "null"]},
                    "amount_mentioned": {"type": ["number", "null"]}
                }
            },
            "requires_human": {"type": "boolean"},
            "confidence": {"type": "number"}
        },
        "required": ["category", "urgency", "requires_human", "confidence"]
    }
}

Entity extraction fails quietly more often than it fails loudly — an invoice number gets read off a forwarded thread instead of the original message, or a dollar figure in a quoted reply is picked up instead of the one in the new text. Test extraction against real historical threads, not clean single-message examples, because threading and quoting is where it breaks.

Deciding whether to draft, route, or hold

Not every category deserves the same level of agent autonomy. Routing a message to the right queue is low-risk and reversible — a misrouted ticket just gets re-routed. Drafting a reply for a human to send is medium-risk. Auto-sending a reply is high-risk, because the moment it goes out you can't unsend it, and a wrong answer to a customer about pricing, a refund, or a contractual term is a real cost. The design decision that matters most here is which category gets which treatment, and it should be a conservative, explicit whitelist rather than something the model decides at runtime.

Guardrails against bad auto-sends

For the narrow set of categories allowed to auto-send (typically "acknowledgment received" style replies, or password-reset-style deterministic responses), add a second, independent check before the send: a regex or classifier that scans the drafted reply for red flags — mentions of specific dollar amounts, promises, legal language, or a tone mismatch with the incoming urgency. If the draft trips any of those checks, downgrade it to human review regardless of the original confidence score. This second pass costs almost nothing and catches the cases where the first model hallucinated a commitment it shouldn't have made.

The review queue and feedback loop

Every drafted-not-sent reply goes into a queue where a human can edit and send, reject, or escalate. Log the diff between what the agent drafted and what the human actually sent — that diff is your evaluation signal. If drafts are consistently getting heavily edited for a particular category, that's a sign the prompt or the underlying knowledge base for that category needs work, not that the model needs to be trusted more.

Latency and inbox volume at scale

A single triage call at a few hundred milliseconds to a couple seconds is fine for interactive use, but a shared inbox with a backlog needs a queue-based worker rather than synchronous processing on email arrival — pull from a queue, classify, write results, move on. Batch the classification calls where the provider supports it if you're processing a large historical backlog; there's no need to pay interactive-latency pricing for triaging a week-old backlog overnight.

Wrapping up

Email triage is a good agent use case because most of the volume is low-stakes and routing mistakes are cheap to fix. The design work is in drawing a hard line around what the agent is allowed to send unattended, backing that line with an independent check rather than trusting the model's own confidence field, and building a review queue whose edit patterns tell you where the system still needs work.