Modern frontier models accept more than text — you can hand them images, and through a transcription step, audio. This unlocks document understanding, visual Q&A, and voice interfaces from a single API. The pitfalls are practical: images are expensive in tokens, quality depends heavily on preprocessing, and sometimes a narrow specialist tool beats the general model.
How image inputs work
You pass an image inline with your text prompt and ask questions about it. Internally the image is converted into tokens, and the count scales with resolution — a large, high-resolution image can cost as much as a long text prompt. So resize and compress to the smallest resolution that preserves the detail you need. Sending a 4000-pixel photo when 1000 pixels would answer the question is a silent cost leak.
Vision model vs OCR
For pulling clean text out of a clear document, a dedicated OCR engine is often faster, cheaper, and more accurate than a vision model — reach for the LLM when you need understanding, not just transcription.
| Task | Better tool |
|---|---|
| Extract text from a clean scan | Dedicated OCR |
| Answer questions about a chart or photo | Vision model |
| Understand a messy form's layout and meaning | Vision model |
| High-volume, structured text extraction | OCR, LLM only for edge cases |
Audio pipelines
Audio is usually a two-step pipeline: transcribe speech to text with a speech-to-text model, then feed the transcript to the LLM for understanding, summarisation, or answering. Keep the steps separate so you can swap or tune each — transcription quality is often the limiting factor, and a bad transcript makes even a great model useless. For voice output, add a text-to-speech step on the response.
Most multimodal problems are input problems. Resize and compress images to the minimum useful resolution, de-noise and segment audio before transcription, and crop to the region of interest. The model can only reason about what you send it clearly — and you pay for every pixel and second you send.
Multimodal features are genuinely capable now, but treat each modality as its own engineering problem: budget image tokens like text tokens, prefer specialist tools where they win, and build audio as a transcribe-then-reason pipeline you can tune stage by stage.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.