Use case · ops-heavy products & document pipelines
Image classification and routing that fails safely
Image classification and routing means deciding, for every image or document that enters your product, which category it belongs to and therefore which workflow, queue, or handler processes it. The stack that usually fits: a prompted vision-language model (VLM) while your categories are still evolving, a custom-trained classifier once the taxonomy is stable and volume is high — and in both cases, per-category confidence thresholds that route uncertain items to a person instead of guessing.
Who this is for
Anywhere intake is a person sorting images
Insurance claims intake sorting damage photos from documents from receipts; support desks triaging screenshots; property managers routing inspection photos; logistics teams separating labels, damage evidence, and proof of delivery; back offices splitting mixed document uploads. The tell is a human whose job is “look at it, then put it in the right pile.”
The team owns an operational metric — time to first action, queue depth, cost per item — and needs routing to be fast, explainable, and safe when unsure. They rarely need exotic recognition; they need a reliable decision with a fallback path, integrated where the images already arrive.
The decision table
Which approach, in which situation
| Your situation | Approach to try first | Why |
|---|---|---|
| Categories still evolving; moderate volume | Prompted vision-language model (VLM) with a fixed label set in the prompt | Category definitions live in language, so changing the taxonomy is a prompt edit — no retraining cycle. |
| Documents dominate (forms, invoices, IDs) | VLM or document-specialist API with structured output | Document routing usually needs a field or two extracted anyway; one call can classify and extract. |
| Stable taxonomy; high volume; tight unit cost | Custom-trained classifier, distilled from an initial VLM-labeled set | Once categories stop moving, a small dedicated model wins on latency and cost per thousand. |
| Some categories are safety-critical | Any of the above, plus per-category confidence thresholds and a human queue | Misroute costs are asymmetric; the design must treat the expensive categories differently. |
Fit conditions, not measurements. The crossover points — where the custom model beats the VLM on your cost curve — only come from testing on your data.
What to measure
Metrics that decide this use case
| Metric | Why it matters here |
|---|---|
| Per-category accuracy | The average hides the one category that misroutes constantly — read the confusion, not the headline number. |
| Misroute cost asymmetry | Sending a claim photo to marketing is annoying; sending a fraud signal to the bulk queue is expensive. Weight categories by consequence. |
| Abstention rate | Share of images the classifier declines to route — the dial between automation rate and error rate. |
| Tail latency | Routing sits in the intake path; p95 decides whether the queue backs up. |
| Cost per thousand items | Under real volume assumptions — the number that decides prompted-VLM versus custom model at scale. |
Design rule: uncertainty is a category
Start small
The first experiment
Export 200–300 recent items with the category a human actually assigned — at natural frequency, so rare categories appear rarely, plus a handful of deliberate junk items whose correct answer is “none of these.” Run the prompted-VLM classifier against that set, compare to the human labels, and read the confusion by category. If the errors cluster where misroutes are cheap, you have a launch candidate with an abstention queue; if they cluster on the expensive categories, you have learned that before it cost anything.
In short
For image routing, start with a prompted vision-language model while categories evolve, graduate to a custom classifier when the taxonomy stabilizes at volume, and always give uncertainty its own queue — routing accuracy only matters per category, weighted by the cost of each misroute.
Related: product image compliance · feasibility testing · glossary