AI Agent for Document Summarisation

Summarisation is the LLM use case with the shortest path to production value and the easiest way to quietly corrupt a decision. A support-ticket digest, a contract clause summary, or a rollup of an ERP audit trail all look the same on the surface — feed in text, get back fewer words — but the failure modes differ from ordinary chat. Wrong numbers in a summary of a finance thread get treated as fact by whoever reads only the summary. This is what actually matters when you build summarisation into a business workflow.

Extractive vs abstractive, and when each is safer

Extractive summarisation picks and reorders sentences that already exist in the source; abstractive summarisation generates new sentences that paraphrase the source. Abstractive summaries read better and compress harder, but every generated sentence is a new opportunity to introduce a claim that was not in the original text. For anything with dollar amounts, dates, or account numbers — AP invoice threads, AR dunning history, GL adjustment notes — bias toward extractive or hybrid approaches: pull the sentences containing the numbers verbatim, and let the model write connective prose around them rather than regenerating the numbers itself.

A practical middle ground is structured extraction first, narrative second: ask the model to pull out entities (amounts, dates, parties, statuses) into a fixed schema, then generate the narrative summary from that schema instead of directly from raw text. If the schema step fails or looks wrong, you catch it before it reaches the reader.

Chunking long documents: map-reduce and its failure modes

An email thread with forty replies, a support ticket with a year of comments, or a contract with fifty pages will exceed a single context window, or at least exceed the point where attention degrades and the model starts skimming. The standard fix is map-reduce: split the document into chunks, summarise each chunk independently (map), then summarise the summaries (reduce). It works, but it has a specific failure mode — information that depends on connecting two facts from different chunks never gets connected, because no single map step sees both. A contract clause defined in section 2 and referenced in section 40 will summarise fine in isolation and lose the cross-reference entirely.

Two mitigations that are worth the extra cost: overlap chunks by a paragraph or two so boundary information survives, and run a final reduce pass with the original key entities (party names, amounts, dates) carried forward explicitly rather than trusting the intermediate summaries to have preserved them.

def summarise_chunk(chunk: str, focus: str) -> str:
    prompt = f"""Summarise the following excerpt in 3-5 sentences.
Focus only on: {focus}
Quote any dollar amounts, dates, or account numbers exactly as written.
If a fact is not stated explicitly, do not infer it.

Excerpt:
{chunk}"""
    return llm.complete(prompt, max_tokens=200)

def reduce_summaries(chunk_summaries: list[str], focus: str) -> str:
    joined = "\n\n".join(chunk_summaries)
    prompt = f"""Combine these partial summaries into one coherent summary,
under 200 words, focused on: {focus}
Preserve every dollar amount and date exactly as given below —
do not recalculate or round.

{joined}"""
    return llm.complete(prompt, max_tokens=300)

Hallucination risk: numbers and dates are where it bites

LLMs summarise prose well and arithmetic poorly. Ask for a summary of an AR aging thread and the model will happily state "the balance is approximately $12,000" when the source says $11,842.50 — close enough to sound right, wrong enough to matter on a reconciliation. The same happens with dates: a model summarising a change log will sometimes collapse "created March 3, last modified March 19" into a single date, or invent a due date that was implied but never stated.

For audit trails specifically, the safest pattern is to treat the LLM as a narrator over a diff you already computed deterministically — you know exactly which fields changed and by how much from the database; the model's job is only to phrase that in plain language, not to re-derive it from unstructured text.

Controlling length and focus with prompts

Vague instructions like "summarise this" produce inconsistent length and inconsistent focus, which is unusable in a UI that needs a predictable card size. Two things fix most of it: give an explicit length constraint tied to a unit the model can count against (sentences, not "briefly"), and give an explicit focus so the model knows what to keep when it has to drop something. "Summarise this support ticket in 3 sentences, focused on unresolved action items" produces a materially different and more useful summary than "summarise this ticket."

For recurring summary types — daily ticket digests, weekly AP exception reports — lock the prompt template down and version it like code. A prompt change that shifts summary length or tone should go through the same review as a schema change, because downstream automation or dashboards may depend on the shape of the output.

Evaluating summary quality: faithfulness before fluency

Fluency is the wrong metric to optimise first — a summary can read beautifully and still misstate the invoice total. The metric that matters for business use is faithfulness: does every factual claim in the summary trace back to something actually present in the source. A cheap, effective check is to extract all numbers and dates from the summary with a regex, then verify each one appears verbatim in the source document; anything that doesn't match is flagged for human review before the summary ships.

For higher-stakes summaries (contract obligations, financial rollups), a second LLM call as a faithfulness checker — "does this summary contain any claim not supported by the source text below, answer yes/no and list them" — catches paraphrased hallucinations the regex check misses, at the cost of a second API call. Reserve it for the summaries someone will actually act on financially, not for every ticket digest.

Wrapping up

Summarisation earns its keep fast because the cost of being wrong looks small — a slightly off sentence, not a broken transaction. That is exactly why it deserves the same discipline as any other production LLM feature: keep numbers out of the model's hands wherever a deterministic value already exists, chunk long documents with overlap so cross-references survive, constrain length and focus explicitly, and check faithfulness before you trust fluency. If you're building this against Acumatica data specifically, reach out or keep reading through the rest of the Acumatica blog.