Cost Monitoring for LLM Apps

LLM bills surprise teams because cost is invisible at the point it is created. A single feature change — a longer system prompt, an extra retrieval chunk, a retry loop — can double token usage without anyone noticing until the monthly invoice. The fix is boring and effective: capture usage on every call, attribute it, and price it in your own system so you can see spend the same day it happens.

Log tokens on every call

Every provider returns input and output token counts in the response. Record them alongside the metadata you will want to slice by later: user or tenant, feature, model, prompt version, and whether the call was a cache hit or a retry. Store it in your own database or metrics system — do not rely on the provider dashboard, which cannot see your application dimensions.

def tracked_complete(messages, *, model, user_id, feature):
    resp = client.messages.create(model=model, messages=messages)
    u = resp.usage
    log_usage(
        user_id=user_id, feature=feature, model=model,
        input_tokens=u.input_tokens,
        output_tokens=u.output_tokens,
        cost_usd=price(model, u.input_tokens, u.output_tokens),
    )
    return resp

def price(model, tin, tout):
    rate = PRICES[model]                 # USD per 1M tokens, from the provider pricing page
    return (tin * rate["in"] + tout * rate["out"]) / 1_000_000

Attribute to features and users

A total spend number tells you there is a problem but not where. The useful questions are per-feature (which capability is expensive) and per-tenant (which customer is unprofitable). Because you logged those dimensions, both are a group-by. This is also what lets you price a product tier honestly: if your heaviest users cost more in tokens than they pay, you found that out in a dashboard instead of a board meeting.

Budgets and alerts

Monitoring without a limit is just a nicer way to watch money leave. Add guardrails:

• A per-tenant or per-user daily/monthly token budget, enforced in the request path — reject or degrade when exceeded rather than logging it after the fact.
• An alert on daily spend crossing a threshold, so a runaway retry loop or a prompt-injection abuse pattern pages you the same day.
• An anomaly check on cost-per-request: a sudden rise usually means a prompt grew or retrieval is returning more context than intended.

The goal is not to minimise spend — it is to make spend a number you can see, attribute, and cap. Once every call is logged with its cost and its owner, cost stops being a monthly surprise and becomes just another metric you tune.