AI Agent for Meeting Summarisation

A meeting summarization agent looks simple until you've shipped one: transcribe, summarize, done. In practice the transcript is the easy input and the hard part is turning an hour of overlapping speech into action items a project manager will actually trust and act on without re-listening to check. The pipeline that holds up in production has distinct stages — diarization, transcription, extraction, and structured output — each with its own failure mode.

Diarization comes before summarization, not after

Speaker diarization — labeling "who said what" — has to happen at the audio-processing stage, not as something you ask the LLM to infer from a wall of undifferentiated text. Whisper's transcription and diarization are separate problems: OpenAI's Whisper API gives you accurate text but no speaker labels, so most pipelines pair it with a dedicated diarization model (pyannote.audio is the common open-source choice) that segments the audio by speaker turn before transcription is aligned to it. Skipping this step and asking the LLM to guess speaker boundaries from text alone produces summaries that misattribute commitments — "I'll own the migration" assigned to the wrong person is the kind of error nobody catches until the wrong person gets chased for it.

From transcript to structured summary

Once you have a speaker-labeled transcript, the summarization call should return structured output, not prose you then re-parse. Use JSON schema mode (OpenAI's response_format or Anthropic's forced tool call) to get a fixed shape back: a short overview, a list of decisions made, a list of action items each with an owner and due-date-if-mentioned, and open questions. This makes the output directly usable by downstream systems — a task tracker, a CRM note, a Slack digest — without another parsing step that can silently drop information.

summary_schema = {
    "name": "summarize_meeting",
    "description": "Produce a structured summary from a diarized transcript",
    "input_schema": {
        "type": "object",
        "properties": {
            "overview": {"type": "string"},
            "decisions": {"type": "array", "items": {"type": "string"}},
            "action_items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "task": {"type": "string"},
                        "owner": {"type": "string"},
                        "due_date": {"type": ["string", "null"]},
                        "source_quote": {"type": "string"}
                    },
                    "required": ["task", "owner", "source_quote"]
                }
            },
            "open_questions": {"type": "array", "items": {"type": "string"}}
        },
        "required": ["overview", "decisions", "action_items"]
    }
}

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=2048,
    tools=[summary_schema],
    tool_choice={"type": "tool", "name": "summarize_meeting"},
    messages=[{"role": "user", "content": diarized_transcript}]
)

The source_quote field matters more than it looks — it's what lets a reviewer trace an extracted action item back to the exact line in the transcript instead of taking the model's word for it.

Long transcripts and context limits

An hour-long meeting can run 8,000-12,000 words of transcript, which fits comfortably in a single call on current frontier models with 200K+ token context windows — you generally don't need to chunk a single meeting. Where chunking does matter is multi-hour workshops or when you're summarizing across a series of related meetings: run per-meeting extraction first, then a second pass that synthesizes across the structured outputs rather than re-feeding raw transcripts. Map-reduce over raw text tends to lose cross-meeting continuity ("this is the third time this blocker came up") that a synthesis pass over structured summaries preserves.

Human review before action items leave the building

Don't auto-send extracted action items straight into a task tracker or a follow-up email. Route the structured summary to the meeting organizer for a quick approve/edit pass first — this is the cheapest guardrail against diarization errors, hallucinated deadlines, or a decision the model inferred that was actually still under discussion. A five-second review click is a small tax against the cost of a wrong commitment landing in someone's inbox with an assigned name attached.

Evaluating summary quality over time

Word-error-rate on transcription is measurable and stable; summary quality is not, and it drifts as meeting formats and topics change. Keep a small set of meetings with human-written "gold" summaries and action-item lists, and periodically score new model outputs against them — an LLM-as-judge rubric checking whether each gold action item was captured, and whether any hallucinated items appeared, catches regressions faster than eyeballing transcripts. Re-run this eval whenever you change the summarization prompt, the schema, or the underlying model.

Wrapping up

A meeting-summary agent is only as good as its weakest stage, and that's usually diarization, not the LLM call everyone focuses on. Get speaker attribution right, force structured output with source quotes for traceability, and keep a human in the loop before action items leave the meeting room — the summarization itself is the part frontier models already handle well.