Guide · Pillar: Choose
Choose the approach, not just the model.
Choose the solution family before any model name. If the output is free text or a judgment over the whole scene, start with a prompted vision-language model (VLM). If you need locations or counts of fixed classes under tight latency, start with a detector. If a specialist API covers exactly your task, test it first. Custom training is a later decision that data and economics must justify. Then test your top two families on about 100 representative images.
Published August 31, 2026 · Pillar: Choose · By the VisionRoute team
Definitions
Model vs stack
The four solution families
Who this guide is for
What the tree returns
The decision tree
Four questions, in order, before any vendor name
Answer with your 12-month volume and your hardest images in mind, not the demo. Terms are defined in the glossary.
| Step | Question to ask | If yes | If no |
|---|---|---|---|
| 1 | Is the output a small, fixed set of answers — labels, fields, or pass/fail verdicts? | Continue to step 2. | Prompted vision-language model (VLM) first. Free-text description, variable attributes, and judgment against written rules sit where language models are strong. Enforce an output schema from day one. |
| 2 | Does a specialist API already cover exactly this task (OCR, barcodes, explicit content, face blurring)? | Specialist API first. It is the fastest path when the task matches — but test it on your hard cases before committing; "exactly" is doing a lot of work in this question. | Continue to step 3. |
| 3 | Do you need object locations or counts, not just an image-level answer? | Pretrained or open-vocabulary detector first. Fine-tune only if your evaluation shows specific slices failing. | Continue to step 4. |
| 4 | Will per-image cost or a strict latency budget bind at your 12-month production volume? | Small custom classifier (or a fine-tuned open model) — if you can label roughly 500–5,000 images. Low marginal cost and stable latency are what you are buying. | Prompted VLM first. Revisit this step when volume grows or the latency budget hardens — the answer is allowed to change. |
Worked example
Walking the tree: marketplace listing photos
A concrete, illustrative walk-through — the marketplace is fictional and nothing here is a measured result. The value is seeing each question do its work.
The product need, in product terms
A marketplace lets sellers upload listing photos. Three rules decide acceptance: plain background, no watermark or overlay text, product fully in frame. About 5,000 photos arrive per day, checked asynchronously at upload.Step 1 — fixed answers? Yes.
Each photo needs three pass/fail verdicts. The output set is small and fixed, so the free-text branch does not apply.Step 2 — an exact specialist API? No.
Text-overlay detection exists as a service, but no single API covers all three rules as this marketplace defines them (its own background and framing standards). Stitching three vendors for one decision adds cost and three failure modes.Step 3 — locations needed? No.
The verdicts are image-level. Framing can be judged without boxes ("does the product touch the image border?"), so a detector is not required by the output.Step 4 — cost or latency binding? Not yet.
Checks run asynchronously, so seconds-scale latency is acceptable. At 5,000 images/day, per-image API pricing is material but not decisive. The tree lands on: prompted VLM, one call per photo, returning JSON verdicts with a confidence per rule.The first experiment
Collect 100 representative photos — including about 30 known rejects and 20 borderline cases — and measure per-rule false accepts and false rejects against the team’s definitions. If volume grows 10×, step 4 flips: move the highest-volume rule to a small classifier and keep the VLM for the long tail.
When this answer changes
A hard latency floor appears
Images cannot leave your infrastructure
The rules change often
Volume grows an order of magnitude
Limitations
What this tree cannot decide
- It covers narrow decisions on still images. Video, robotics, edge hardware, medical, and biometric applications have constraints this tree does not model.
- It returns a starting family, not a verdict. Two families that both survive the tree are separated by testing on your images — not by further reading.
- It contains no benchmark numbers by design: latency and cost claims here are directional, and the honest figures are the ones you measure under your own serving path and volume.
In short
Pick a computer-vision approach with four ordered questions: Is the output a fixed set of answers? Does a specialist API cover exactly this task? Are locations or counts required? Will cost or latency bind at production volume? Start with the family the tree returns, and let roughly 100 representative images make the final call.
Related: VLM vs object detection · custom model vs hosted API · feasibility testing