Machine learning runs on expensive hardware — GPUs and accelerators — and the costs escalate quietly. Optimising them starts with a distinction that decides where to focus: cost splits into training (building models) and inference (serving them). Early on, training dominates; at scale, inference — running the model on every request, forever — usually dwarfs it. Knowing which is your real cost is the first optimisation.
Attribute the spend first
You cannot optimise what you cannot see, so break the bill into training versus inference, and within each, by project and model. This attribution almost always surprises teams — often a single model's inference, or a set of experiments nobody remembers, dominates. As with any cost work, the Pareto rule holds: a small number of things drive most of the spend, and finding them is most of the win.
Cutting training cost
- Use spot / preemptible instances — training is interruptible and restartable from checkpoints, so it is ideal for the much cheaper spot market.
- Right-size the hardware — do not train on the biggest GPU by default; match it to the model, and profile utilisation.
- Stop wasteful runs early — kill experiments that are clearly not converging instead of letting them finish.
- Use efficient methods — techniques like parameter-efficient fine-tuning train far less than a full run for comparable results.
Cutting the inference bill
At scale, inference is where the money goes, because you pay for it on every prediction indefinitely — and it is where optimisation compounds. Smaller or distilled models that match a larger one's quality on your task cut per-prediction cost directly. Batching predictions improves hardware utilisation. Caching repeated predictions avoids recomputation entirely. And crucially, serving pattern matters: a workload built as a real-time API that could have been batch is paying for latency it never needed. The inference bill responds to the same efficiency levers as any high-volume system, multiplied by ML's hardware cost.
Teams instinctively optimise training because that is where ML feels expensive, but a model in production serves predictions forever, so inference cost accumulates without end. If you are past the experimentation stage and serving real traffic, your biggest lever is almost certainly inference — a smaller model, batching, caching, or the right serving pattern — not shaving hours off training runs.
ML cost optimisation begins with attributing spend across training and inference, then applying the right levers to each: spot instances, right-sized hardware, and efficient methods for training; smaller models, batching, caching, and the correct serving pattern for inference. Focus follows the stage — training early, inference at scale — and since a production model serves predictions indefinitely, the inference bill is usually where the durable savings live.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.