A code review agent that reads a diff and leaves comments is easy to demo and easy to make annoying in practice. The useful version is scoped narrowly: it flags a specific class of issue it can check reliably — missing null checks around ERP field access, unbounded queries, hardcoded credentials, SQL built by string concatenation — rather than trying to be a generalist reviewer competing with a human's judgment on architecture and naming.
Scoping the review to checkable claims
The failure mode of a general-purpose "review this PR" prompt is that half its comments are style opinions dressed up as findings, and the useful ones get lost in the noise. Better to run several narrow, tool-backed passes: one that greps the diff for known-risky patterns and asks the model to confirm or dismiss each match with the surrounding context, rather than one broad pass asking the model to find everything wrong. A finding the model can't tie to a specific line and a specific reason gets dropped before it reaches a human.
def flag_finding(file: str, line: int, severity: str, rule: str, explanation: str) -> None:
"""Called by the model for each concrete issue found.
severity: 'blocking' | 'suggestion'
rule: the specific check this maps to, e.g. 'unbounded-query'
"""
...
# The model is given the diff plus static-analysis output
# (e.g. a query without a WHERE clause, a secret-looking string)
# and asked only to confirm/reject each candidate, with reasoning.Pairing with static analysis, not replacing it
Don't ask the LLM to find every SQL injection risk in a diff from scratch — that's a search problem a linter or static analyzer does deterministically and cheaply. Run the deterministic scanner first, then hand the LLM the candidate matches plus surrounding code and ask it to judge false positives and add human-readable context. This division of labor plays to each tool's strength: the scanner never misses a pattern it's configured to catch, and the LLM is good at judging whether a matched pattern is actually exploitable given the surrounding logic.
Static analyzers are noisy — a raw scan of a large codebase produces hundreds of matches, most irrelevant. The agent's job is filtering that list down to the handful worth a human's attention, with a plain-language reason for each.
Keeping the diff inside the context budget
Large PRs blow past what's useful to hand an LLM in one call — both because of token limits and because review quality degrades on very long inputs. Chunk by file, and for each file, include only the changed hunks plus a few lines of surrounding context rather than the whole file, unless a specific check (like tracing a variable's origin) requires more. For genuinely large refactors, review file-by-file and aggregate findings afterward rather than trying to reason about the entire diff in a single pass.
Why false positives are the metric that matters
A code review bot that's right 90% of the time but wrong loudly enough to annoy developers gets muted within a week — engineers stop reading its comments, which makes the whole system worse than nothing. Track the dismissal rate on flagged findings as your primary quality metric, not raw recall. If reviewers are dismissing more than roughly 1 in 5 findings, tighten the rule set or raise the confidence bar before adding new checks, since credibility is what keeps the bot's remaining comments getting read.
Scope the bot's authority to leaving comments and setting a "needs human review" label. Merge authority stays with people — an agent auto-approving code is a liability the moment its blind spot lines up with an actual vulnerability.
Treat it like a classifier, and evaluate it like one
Build a small labeled set of past PRs with known issues (bugs that shipped, security findings from a real audit) and known clean diffs, and run the agent against both. Track precision (of the findings it raised, how many were real) and recall (of the known issues, how many it caught) separately, and re-run this eval any time you change the prompt, the model version, or the rule set — a silent regression here erodes trust fast and is hard to notice from spot-checking alone.
| Check type | Best handled by |
|---|---|
| Known-pattern security issues | Static analyzer, LLM triages false positives |
| "Is this logic actually correct given context" | LLM, given the specific hunk and surrounding code |
| Style/formatting | Linter — don't waste LLM calls on this |
| Architecture/design judgment | Human reviewer, not the agent |
Wrapping up
The agents that survive in a code review pipeline are the ones with a narrow, well-defined job and a low false-positive rate, not the ones that try to reason about the whole PR like a senior engineer would. Pair it with the static analysis you already trust, measure precision relentlessly, and keep merge authority with people.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.