Fine-tuning adapts a base model to your task by continuing its training on your examples. Done for the right reason it delivers consistent behaviour, shorter prompts, and a cheaper model that punches above its size on a narrow task. Done reflexively, it is an expensive way to bake yesterday's data into frozen weights. The patterns below are what separate the two.
Confirm it is the right tool first
Fine-tune for behaviour — a fixed output format, a consistent voice, a specialised classification — not for knowledge that changes, which belongs in RAG. And only after prompting and retrieval have hit their ceiling, because fine-tuning is a slower, costlier iteration loop. If the requirement will keep changing, you are signing up to retrain every time.
Data is the whole game
The result is a direct function of the training data. Invest there:
- Quality over quantity — a few hundred clean, correct, consistent examples beat thousands of noisy ones; the model imitates whatever you show it, including the mistakes.
- Match production — training inputs should look like real traffic, or the model learns a distribution it will never see.
- Version the dataset — treat it like code; you cannot reproduce or debug a model without knowing exactly what it trained on.
- Hold out a test split — keep evaluation examples the model never sees during training.
LoRA / PEFT
You rarely need to retrain all the weights. Parameter-efficient methods like LoRA train a small set of adapter weights on top of a frozen base model — far cheaper and faster, with results competitive with full fine-tuning for most adaptation tasks. Adapters are also small and swappable, so you can keep several task-specific adapters over one base model instead of maintaining many full copies.
Avoid catastrophic forgetting
Over-train narrowly and the model gets worse at everything else — it forgets general capability while learning your task. Guard against it: use modest learning rates and few epochs, keep some general examples in the mix, and always evaluate on broad capability as well as your target task so you notice the regression before you ship it.
The only question that matters is whether the fine-tuned model beats the base model plus good prompting on your eval set. Measure both on the same held-out data. Fine-tuning that wins on the training distribution but loses on real inputs — or costs more than it saves — is a project you should not ship.
Approach fine-tuning as a data-engineering discipline: prove prompting and RAG are exhausted, curate and version a small high-quality dataset, prefer LoRA for cost and flexibility, watch for forgetting, and gate the whole thing on an honest comparison to the base model. The model training is a button; the data and the evaluation are the work.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.