Training a model is only useful once it makes predictions in production, and there are several ways to serve it — each suited to a different need. The mistake is defaulting to a real-time API for everything, including workloads that would be far cheaper and simpler as a batch job. Choosing the serving pattern starts with one question: when and how fast do you need the predictions?
The serving patterns
| Pattern | Predictions | Fits |
|---|---|---|
| Online / real-time | On demand, low latency, one at a time | Interactive features — recommendations, fraud checks at request time |
| Batch | Many at once, on a schedule | Predictions that aren't needed instantly — nightly scoring, reports |
| Streaming | Continuously on an event stream | Real-time event processing — live anomaly detection |
Online serving exposes the model behind an API for immediate, per-request predictions — necessary when a user is waiting. Batch runs the model over a large dataset on a schedule, which is dramatically cheaper and simpler when the predictions are not needed the instant the data arrives. A huge amount of ML in production is batch, and mistakenly building it as a real-time service adds cost and operational burden for latency nobody needed.
Online serving considerations
If you do need real-time, online serving brings the concerns of any low-latency service: latency targets, autoscaling to handle load, and keeping the service available. The model runs behind an endpoint that must respond within your latency budget and scale with traffic. This is the most operationally demanding option, which is exactly why you should confirm you truly need per-request latency before choosing it over batch.
Build versus managed
Whichever pattern, you decide whether to build the serving infrastructure yourself or use a managed serving platform. Managed platforms handle scaling, deployment, and monitoring for you, trading some control and cost for far less operational work — often the right call unless you have specific requirements or scale that justify building. Rolling your own serving stack is real, ongoing infrastructure work; be sure the need justifies it before taking it on.
The default should be the simplest pattern that meets the need, and that is usually batch. Real-time online serving is more expensive and more operationally demanding, so reserve it for predictions genuinely needed at request time. A great deal of production ML that gets built as a real-time API would have been cheaper, simpler, and just as effective as a scheduled batch job.
Model serving options — online for low-latency per-request predictions, batch for scheduled bulk scoring, streaming for continuous event processing — should be matched to whether predictions are needed instantly or can wait, with a build-versus-managed decision layered on top. Default to batch and managed for simplicity and cost, and step up to real-time, self-built serving only when the latency and control requirements genuinely demand it.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.