Data Leak Prevention in LLM Apps

An LLM agent wired to your ERP has, by construction, broad read access to customer data, pricing, and financials — and every tool call result becomes part of the context sent to a third-party API. Data leak prevention here means two separate concerns: making sure the agent doesn't retrieve more than the current user is entitled to see, and making sure sensitive fields don't end up somewhere they shouldn't — provider logs, a wrong recipient, a downstream system without the same access controls.

Permission checks belong in the tool, not the prompt

"Only show data the current user is authorized to see" as a system prompt instruction is not a security boundary — a sufficiently unusual request, an injected instruction, or just model inconsistency can bypass it. Every tool that reads data should independently verify the requesting user's permissions against the record being accessed, the same way you'd guard a REST endpoint, regardless of what the prompt says or what the model claims about who's asking.

def get_customer_record(customer_id: str, requesting_user: User) -> dict:
    """The permission check happens here, in code,
    every time — never delegated to the model's judgment."""
    if not requesting_user.can_view(customer_id):
        raise PermissionDenied(f"{requesting_user.id} cannot view {customer_id}")
    record = db.fetch_customer(customer_id)
    return redact_fields(record, requesting_user.clearance_level)

Redact before the data reaches the model, not after

If a field shouldn't be visible to a given user, don't fetch it and trust the model not to repeat it — strip it at the data layer before it enters the prompt. A model can't leak a social security number, a full card number, or an internal margin figure it was never given. This also shrinks your token footprint, since fields the current request doesn't need don't need to be fetched or redacted at all — just excluded from the query.

What happens to the data after the API call

Understand your LLM provider's data retention and training-use policies before sending anything sensitive — Anthropic's and OpenAI's enterprise/API tiers generally don't train on API data by default, but logging and retention windows still apply, and your own application logs are usually the bigger risk. If your app logs full prompts and responses for debugging, that log store now contains whatever customer data passed through the LLM call, and it needs the same access controls and retention policy as your production database, not the defaults your logging framework ships with.

Cross-tenant isolation when the agent serves multiple customers

If one agent deployment serves multiple tenants (common in a vertical SaaS or multi-client consulting setup), retrieval and tool calls need tenant ID scoping enforced at the query level, not just implied by which agent instance is running. A shared vector store queried without a tenant filter, or a tool that accepts a record ID without checking it belongs to the caller's tenant, is a cross-tenant data leak waiting for the wrong combination of inputs.

Testing for leaks as part of the eval suite, not just hoping

Build test cases that deliberately probe for leakage: a user requesting data belonging to another account, a request phrased to coax the model into revealing a redacted field, a prompt injection embedded in retrieved content trying to exfiltrate context. Run these against every prompt or model change, the same way you'd run a regression suite — a permission boundary that held last month can silently break after a seemingly unrelated prompt tweak.

Wrapping up

Data leak prevention for an LLM-integrated agent is mostly the same discipline as securing any API that touches sensitive data — permission checks in code, redaction before data leaves your boundary, careful logging — plus the reminder that a prompt instruction is not a security control. Test for leaks deliberately, because a subtle regression here won't announce itself.