An LLM agent without memory re-derives everything from scratch on every call: who the user is, what they asked five minutes ago, which invoice they were disputing. That is fine for a single-turn Q&A bot and useless for anything that runs a multi-step workflow against an ERP. Memory is what turns a stateless completion API into something that can carry a task across turns, tool calls, and — if you design it right — across sessions entirely.
The three kinds of memory that matter
It helps to stop treating "memory" as one thing. In practice you're managing three separate stores with different lifetimes. Working memory is the current conversation or task — the message history you feed back into the context window on every call, bounded by token budget. Episodic memory is a record of past interactions — "last Tuesday this customer asked about a credit memo on invoice INV-004821" — retrieved by similarity search when relevant, not loaded wholesale. Semantic memory is durable facts that don't change turn to turn: this customer's preferred currency, their credit terms, the fact that they always want PO numbers on the invoice line. Conflating these is the single most common design mistake — teams dump everything into one vector store and wonder why retrieval gets noisy.
Working memory is just context-window management
For the current task, the discipline is pruning, not storage. Full conversation transcripts grow linearly and burn tokens on turns that no longer matter. A workable pattern: keep the last N turns verbatim, and replace anything older with a running summary generated by a cheap, separate LLM call. When the agent calls a tool — say, a lookup against Acumatica's contract-based REST API for a sales order — store the tool call and its result as structured data, not prose, so a later summarization pass can compress it without losing the order number or amount.
If a number matters — an invoice total, a quantity, a date — do not let it live only inside a summarized blob. Pull anything the workflow depends on downstream into structured state (a plain dictionary or a row in your own database) that isn't subject to the LLM's compression.
Episodic recall with a vector store
For "has this come up before," embed the interaction (user request + resolution) and store it with metadata — customer ID, date, entity type — in a vector database. Retrieval should always be filtered by metadata first and ranked by similarity second; a semantic match against the wrong customer's history is worse than no match. Keep the embedded text short and factual rather than including the full back-and-forth — embed the outcome, not the negotiation.
def recall_similar_cases(customer_id: str, query: str, k: int = 3):
results = vector_store.query(
vector=embed(query),
filter={"customer_id": customer_id, "type": "resolved_case"},
top_k=k,
)
# Only feed the agent a compact summary, not raw transcripts
return [
f"{r.metadata['date']}: {r.metadata['summary']}"
for r in results if r.score > 0.78
]Semantic memory belongs in your database, not the prompt
Stable facts about an entity — a customer's tax exemption status, a vendor's default payment terms, a branch's fiscal calendar — should not be re-extracted by the LLM from old conversations. They should be columns in a table, fetched with a normal query and injected into the system prompt as structured facts. Treating durable business data as something the LLM "remembers" from chat history is fragile: it's one bad extraction away from quoting a stale credit limit. Let the ERP be the source of truth and let memory systems handle only what genuinely doesn't have a canonical home yet.
Forgetting is a feature
Unbounded memory growth degrades retrieval quality — more candidates, more near-duplicates, more chances the wrong one gets pulled. Decide on a retention policy up front: episodic entries older than a defined window get archived out of the live index, and anything containing PII gets a separate, shorter retention aligned to your privacy obligations (see the compliance discussion elsewhere on this site for GDPR/POPIA specifics). A memory system that never forgets isn't more capable, it's just slower and noisier.
| Memory type | Where it lives | Lifetime |
|---|---|---|
| Working (current task) | Context window, pruned/summarized | Single session |
| Episodic (past cases) | Vector store, metadata-filtered | Weeks to months, with expiry |
| Semantic (stable facts) | Relational database | Until the business fact changes |
Wrapping up
Most memory problems in production agents are retrieval problems wearing a memory costume — the fix is usually better metadata filtering, not a bigger context window or a fancier embedding model. Keep working memory pruned, keep episodic recall filtered and scored, and keep anything the business already tracks in its system of record rather than asking the LLM to remember it. If you're building this against Acumatica specifically, the API already gives you the semantic layer for free — use it instead of reinventing it in a vector store.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.