Project risk detection is the ai-agent use case with the shortest path to trust, because the output is a flag on a dashboard, not a transaction. A model that misjudges a project's health wastes a project manager's ten minutes; it does not post a bad invoice. That lower blast radius is exactly why it's a good place to start experimenting with agents on top of ERP data, and it's still worth building carefully, because a system that cries wolf gets ignored within a month.
What signals actually predict trouble
The useful signals are almost all structural, not textual. Burn rate against budget, the ratio of billed-to-date versus percent-complete on a percentage-of-completion project, the drift between original committed cost and current committed cost, the age of the last actual-cost entry, and change order volume relative to original contract value. These live in the Projects module — PM Project (PM301000), the project balances tables, and the cost/revenue budget lines. None of this needs an LLM to compute; a SQL query over PMProject, PMBudget, and PMTran gets you the numbers.
Where an LLM earns its place is turning six drifting numbers into one sentence a project manager will actually read, and doing it across fifty projects instead of one, on a schedule, without someone opening each project card. The model's job is synthesis and prioritization, not calculation.
The pipeline shape
The pipeline is deterministic data extraction, then a single LLM summarization pass, then deterministic delivery. Pull project financials on a nightly job, compute the risk indicators in code (cost variance percent, schedule variance, days since last time entry), and only pass the computed indicators — not raw transaction dumps — into the prompt. This keeps the context small, keeps the cost predictable, and means the model is reasoning over numbers you already trust rather than re-deriving them from scratch, which is where hallucinated totals creep in.
Compute variance, burn rate, and schedule slippage in code. Pass the computed numbers to the model and ask it to rank and explain. If the model is asked to do arithmetic on a table of transactions, expect wrong totals some fraction of the time — and you won't know which fraction until an auditor asks.
Thresholds before language
Before any text generation happens, a rules layer buckets each project into a risk tier using thresholds a controller would recognize: cost variance beyond 10%, no time entries in 14 days, budget consumed faster than percent-complete. The LLM is only invoked for projects that cross a tier boundary, and its job is narrowly scoped — explain why this project tripped the rule, in two sentences, referencing the specific numbers. This keeps false positives bounded by the same logic a human reviewer would apply, and it keeps the LLM from inventing risk factors that aren't in the data.
SYSTEM = """You summarize project risk for a PM audience.
Rules:
- Only reference numbers present in the input JSON.
- Do not invent causes; describe the variance, not why it happened.
- Two sentences maximum. No recommendations, just the observation.
"""
def summarize_risk(project: dict) -> str:
# project = {"id": "PR-00231", "cost_variance_pct": 14.2,
# "pct_complete": 55, "pct_billed": 38,
# "days_since_last_entry": 21}
user_prompt = json.dumps(project)
return llm.complete(system=SYSTEM, user=user_prompt, max_tokens=120)False positives are the real cost
The failure mode that kills these systems isn't a missed risk — it's noise. If a PM gets flagged on a project that's actually fine because a milestone invoice hasn't posted yet, they stop trusting the flag, and then the system is dead weight. Tune thresholds against six months of closed projects before rollout: for each historical project, check whether your rule would have fired, and compare against whether the project actually ran over budget or late. If your false-positive rate is above roughly one in five flags, the thresholds are too tight and need loosening before anyone sees the output.
Where this fits organizationally
Ship this as a weekly digest, not a real-time alert. Project risk doesn't change hour to hour, and a real-time channel trains people to ignore it the same way they ignore a chatty CI bot. A Monday-morning email or Teams message summarizing the three or four projects that moved into a higher risk tier since last week, each with the one-sentence explanation, is a format PMs will actually open. Route it through the same business-events mechanism already used for other Acumatica notifications rather than building a separate delivery path.
| Layer | Responsibility |
|---|---|
| Nightly extract | Pull PMProject, PMBudget, PMTran into computed indicators |
| Rules engine | Bucket into risk tiers using fixed thresholds |
| LLM | Explain the flagged variance in plain language, nothing else |
| Delivery | Weekly digest to PMs, not real-time alerts |
Wrapping up
The pattern that works is boring on purpose: compute the numbers in code, gate on fixed thresholds, and use the model only to turn a handful of trusted figures into a sentence a busy PM will read. Resist the urge to let the LLM see raw transactions or decide what counts as risky — that's a rules problem, not a language problem, and treating it as one is how these projects end up ignored within a quarter.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.