Chain-of-thought (CoT) prompting asks the model to work through a problem in steps before giving its answer, rather than jumping straight to a conclusion. On problems that require several linked inferences, this reliably improves accuracy, because the intermediate steps give the model room to reason instead of guessing in one shot. But it is not free, and it is not always warranted.
When it helps and when it doesn't
CoT pays off on multi-step tasks: arithmetic and math word problems, logical deduction, planning, anything where the answer depends on a chain of sub-conclusions. It does little for simple lookups, classification, or extraction — there the reasoning is padding that adds latency and cost without changing the answer. Match the technique to the problem's actual depth.
Structuring the reasoning
A plain "think step by step" works, but structure helps more: ask for numbered steps, or a specific scratchpad format, and then a clearly delimited final answer. The delimiter matters for parsing — you want to extract the conclusion reliably without dragging the reasoning into your output.
Work through the problem step by step inside <reasoning> tags.
Then give only the final result inside <answer> tags.
<reasoning>
1. ...
2. ...
</reasoning>
<answer>42</answer>Show or hide the reasoning
The reasoning is a means, not usually the product. Hide it from end users unless the explanation itself is the value (a tutor, an audit trail). Keep it server-side, parse out the final answer, and optionally log the reasoning for debugging. Exposing raw chains of thought also risks leaking prompt details and confusing users with the model's occasional wrong turns.
Dedicated reasoning models — OpenAI's o-series, Claude's extended thinking — do this internally and are trained for it, so manually prompting elaborate chains of thought on top of them is often redundant or counterproductive. On those models, state the problem clearly and let the model allocate its own thinking; save hand-written CoT for standard models on genuinely multi-step tasks.
Use CoT deliberately: on hard, multi-step problems with a standard model it is a real accuracy lever; on simple tasks it is wasted tokens; and on reasoning models it is mostly already handled for you. The skill is not writing longer prompts — it is knowing which of those three situations you are in.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.