Red Teaming LLM Apps

Red teaming an LLM app means deliberately trying to break it before someone with worse intentions does, and it's a different exercise from functional testing. Functional tests check that the agent does what it's supposed to; red teaming checks what it does when someone actively tries to make it do something else. Both are needed, and skipping the second because the first passes is how avoidable incidents happen.

Build an attack catalog, not a vibe check

Ad hoc "let's try to jailbreak it" sessions find some issues, but they don't repeat and they don't scale across releases. Maintain a structured catalog of attack categories instead: role-override attempts, indirect injection via retrieved content (a support ticket or vendor note containing embedded instructions), data exfiltration attempts (getting the model to reveal other customers' records or system prompts), and privilege escalation (getting a low-permission user's session to trigger a high-permission action). Each category gets a set of concrete test cases with expected safe behavior, run against every release the same way a regression suite is.

The highest-value target is the tool boundary

For ERP agents specifically, the interesting attacks aren't really about getting the model to say something embarrassing — they're about getting it to call a tool it shouldn't, with parameters it shouldn't have. Test whether a crafted input can get the agent to call ApproveInvoice on an invoice outside the user's approval limit, or to read a customer record outside the user's branch access. If the tool layer itself enforces permissions correctly, most prompt-level attacks become irrelevant; if it doesn't, no amount of prompt hardening will save you.

Use a second model as an adversary

Manually writing every attack variant doesn't scale. Use a separate LLM instance, prompted explicitly to attempt to break the target system's stated constraints, to generate candidate attack inputs at volume, then have a human review which ones actually succeeded. This surfaces phrasings and framings a human red-teamer wouldn't think to try, and it's cheap to run repeatedly as the target prompt evolves.

ATTACK_CATEGORIES = ["role_override", "indirect_injection", "data_exfil", "privilege_escalation"]

def run_red_team_pass(target_agent, low_priv_session):
    results = []
    for category in ATTACK_CATEGORIES:
        for attack in load_attack_cases(category):
            response = target_agent.run(attack.input, session=low_priv_session)
            passed = attack.safe_behavior_check(response)
            results.append({"category": category, "id": attack.id, "passed": passed})
    failures = [r for r in results if not r["passed"]]
    if failures:
        raise RedTeamFailure(f"{len(failures)} red-team cases failed", failures)
    return results

Track severity, not just pass/fail

Not every successful attack is equally bad. Getting the model to use informal language when asked not to is a minor finding; getting it to disclose another customer's AR balance is a blocking one. Score each finding by what an attacker could actually achieve — data exposure, unauthorized transaction, privilege escalation — and gate releases on the severe category, not on a raw pass percentage that treats a tone violation the same as a data leak.

Repeat it on every material change

A red team pass that ran once before launch tells you nothing about the system six months later, after the system prompt has been edited a dozen times and three new tools have been added. Re-run the full attack catalog whenever the system prompt, the tool surface, or the underlying model changes — model upgrades in particular can silently shift behavior around edge cases that used to be safely refused.

Wrapping up

Red teaming an ERP agent is worth doing as a repeatable, cataloged process, not a one-off exercise — focus effort on the tool boundary where real damage happens, always test under the lowest-privilege account, use an adversarial model to generate volume, and gate releases on severity rather than a raw pass rate.