Running an open-weight model yourself is a genuinely different trade-off than calling a hosted API, not a cheaper clone of the same thing. You give up the frontier-model ceiling and the zero-ops convenience, and in exchange you get fixed cost per token, data that never leaves your network, and a model you can quantize, fine-tune, or pin to a specific version indefinitely. For agent workloads — tool use, structured output, long context — the local story has closed a lot of ground but still has sharp edges worth knowing before you commit infrastructure to it.
Picking a model and quantization level
The current Llama-family and comparable open-weight models (Llama 3.x, Mistral/Mixtral, Qwen2.5) ship at a range of parameter counts, and the parameter count you can actually run is bounded by VRAM, not ambition. A rough rule: full fp16 needs about 2 GB of VRAM per billion parameters, 8-bit quantization roughly halves that, and 4-bit (GPTQ, AWQ, or GGUF's Q4_K_M) gets you to about 0.6-0.7 GB per billion parameters with a modest, usually acceptable quality drop. A 70B model at 4-bit fits in roughly 40-48 GB of VRAM — a single high-end workstation card or a couple of consumer cards — where fp16 would need multiple data-center GPUs.
Quantization degrades tool-calling reliability before it degrades fluent prose; a model that reads fine in chat can start emitting malformed JSON or hallucinated function names at 4-bit under load. If your agent depends on structured tool calls, test the exact quantized checkpoint against a tool-use eval set before assuming the quality numbers from a general benchmark transfer.
Tool-use support is uneven, and it matters most
Hosted APIs guarantee schema-conformant function calling because the provider trains and constrains for it server-side. Local serving stacks have to replicate that themselves. llama.cpp and vLLM both support grammar-constrained decoding — GBNF grammars or JSON-schema-constrained sampling — which forces the model's output tokens to match a schema at the sampling level, not just via prompting. Use that constrained decoding rather than relying on prompted "respond only in JSON" instructions; unconstrained local models drift into prose or broken JSON far more often than hosted GPT-4o or Claude with native structured output modes.
# Serve an OpenAI-compatible endpoint with guided decoding enabled
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--quantization awq \
--max-model-len 8192 \
--guided-decoding-backend outlines
# Client call — schema is enforced at the token level, not hoped for
curl http://localhost:8000/v1/chat/completions \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Extract the order ID and status"}],
"guided_json": {"type":"object","properties":{"order_id":{"type":"string"},"status":{"type":"string"}}}
}'Deployment: llama.cpp, vLLM, or a managed inference box
For single-node, low-concurrency workloads — an internal tool, a batch job — llama.cpp with a GGUF quant is the least operational overhead: one binary, CPU offload for layers that don't fit in VRAM, no Python dependency hell. For anything serving concurrent agent sessions, vLLM's continuous batching and PagedAttention give you real throughput gains, at the cost of running an actual inference server you now own the uptime of. Neither replaces the retry logic, rate limiting, and circuit breakers you'd get for free from a hosted API — you have to build those yourself, because your own GPU box has no SLA.
A hosted provider absorbs GPU failures, model upgrades, and capacity scaling invisibly. Self-hosting means you now own health checks, autoscaling (or accepting a fixed ceiling), driver and CUDA version drift, and the on-call rotation for when the inference box falls over at 2am. Budget for this before comparing per-token cost — the cost that matters is total cost of ownership, not GPU-hour price.
When local actually wins
Local makes sense when: data residency or compliance rules out sending prompts to a third party, request volume is high and steady enough that fixed GPU cost beats per-token pricing, or latency needs to be sub-100ms and colocated with the data. It does not make sense when your traffic is bursty, your team doesn't want to own GPU infrastructure, or your agent's hardest tasks need frontier-level reasoning that open-weight models at any quantization still lag behind on — long-horizon planning and complex multi-step tool orchestration are still where hosted frontier models pull ahead.
| Factor | Hosted API | Self-hosted open-weight |
|---|---|---|
| Cost model | Per-token, scales with usage | Fixed GPU cost, scales with capacity |
| Data residency | Leaves your network | Stays on-prem/VPC |
| Tool-call reliability | High, provider-constrained | Depends on grammar-constrained decoding |
| Ops burden | None — provider's problem | Yours — GPUs, drivers, scaling |
| Frontier reasoning | Best available | Lags on hard multi-step tasks |
Wrapping up
Self-hosting a Llama-family model for agent workloads is a real option in 2026, not a compromise you settle for — but only once you've priced in the GPU ops you're taking on and validated tool-call reliability at your chosen quantization level with grammar-constrained decoding. Start with the smallest model that clears your tool-use eval bar; a well-quantized 8B model with constrained decoding often beats a poorly-served 70B model in practice.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.