VISIONROUTE

Use case · ops-heavy products & document pipelines

Image classification and routing that fails safely

Image classification and routing means deciding, for every image or document that enters your product, which category it belongs to and therefore which workflow, queue, or handler processes it. The stack that usually fits: a prompted vision-language model (VLM) while your categories are still evolving, a custom-trained classifier once the taxonomy is stable and volume is high — and in both cases, per-category confidence thresholds that route uncertain items to a person instead of guessing.

Who this is for

Anywhere intake is a person sorting images

Insurance claims intake sorting damage photos from documents from receipts; support desks triaging screenshots; property managers routing inspection photos; logistics teams separating labels, damage evidence, and proof of delivery; back offices splitting mixed document uploads. The tell is a human whose job is “look at it, then put it in the right pile.”

The team owns an operational metric — time to first action, queue depth, cost per item — and needs routing to be fast, explainable, and safe when unsure. They rarely need exotic recognition; they need a reliable decision with a fallback path, integrated where the images already arrive.

The decision table

Which approach, in which situation

Your situationApproach to try firstWhy
Categories still evolving; moderate volumePrompted vision-language model (VLM) with a fixed label set in the promptCategory definitions live in language, so changing the taxonomy is a prompt edit — no retraining cycle.
Documents dominate (forms, invoices, IDs)VLM or document-specialist API with structured outputDocument routing usually needs a field or two extracted anyway; one call can classify and extract.
Stable taxonomy; high volume; tight unit costCustom-trained classifier, distilled from an initial VLM-labeled setOnce categories stop moving, a small dedicated model wins on latency and cost per thousand.
Some categories are safety-criticalAny of the above, plus per-category confidence thresholds and a human queueMisroute costs are asymmetric; the design must treat the expensive categories differently.

Fit conditions, not measurements. The crossover points — where the custom model beats the VLM on your cost curve — only come from testing on your data.

What to measure

Metrics that decide this use case

MetricWhy it matters here
Per-category accuracyThe average hides the one category that misroutes constantly — read the confusion, not the headline number.
Misroute cost asymmetrySending a claim photo to marketing is annoying; sending a fraud signal to the bulk queue is expensive. Weight categories by consequence.
Abstention rateShare of images the classifier declines to route — the dial between automation rate and error rate.
Tail latencyRouting sits in the intake path; p95 decides whether the queue backs up.
Cost per thousand itemsUnder real volume assumptions — the number that decides prompted-VLM versus custom model at scale.

Design rule: uncertainty is a category

The strongest routing systems treat “I am not sure” as a first-class output with its own queue. An automation rate of 80% with trustworthy confidence beats 100% with silent misroutes — the remaining 20% is exactly the work your team should see.

Start small

The first experiment

Export 200–300 recent items with the category a human actually assigned — at natural frequency, so rare categories appear rarely, plus a handful of deliberate junk items whose correct answer is “none of these.” Run the prompted-VLM classifier against that set, compare to the human labels, and read the confusion by category. If the errors cluster where misroutes are cheap, you have a launch candidate with an abstention queue; if they cluster on the expensive categories, you have learned that before it cost anything.

In short

For image routing, start with a prompted vision-language model while categories evolve, graduate to a custom classifier when the taxonomy stabilizes at volume, and always give uncertainty its own queue — routing accuracy only matters per category, weighted by the cost of each misroute.

Related: product image compliance · feasibility testing · glossary