Guide · Pillar: Choose
Custom model or hosted API? Price the whole stack.
Start with a hosted API or prompted VLM when it covers your task and volume is modest: you pay per call and skip training data entirely. Move to a custom model when one of four break points arrives — unit economics at volume, a hard latency floor, data that cannot leave your infrastructure, or accuracy gaps on your slices that prompting cannot close. Many teams end up hybrid, and that is a result, not a compromise.
Published August 31, 2026 · Pillar: Choose · By the VisionRoute team
Definitions
Hosted vision API
Custom-trained model
Who this guide is for
The question, precisely
The cost-and-control matrix
What each path costs, beyond the per-image price
No prices are quoted deliberately: provider pricing changes and break-even depends on your volume and the engineering cost you assign. Fill the numbers in with your own assumptions — dated — before deciding. Definitions are in the glossary.
| Dimension | Hosted API / VLM | Custom-trained model | Hybrid |
|---|---|---|---|
| Time to first result | Hours to days: integrate, prompt, threshold. | Weeks: collect, label, train, serve, evaluate. | Hours for the API path; custom follows where volume concentrates. |
| Upfront cost | Integration effort only. | Labeling, training runs, serving setup, and an evaluation harness. | Staged — custom costs are paid only for the slice that earns them. |
| Marginal cost per image | The per-call price, forever; grows linearly with volume. | Low once served; you pay for capacity instead, including its idle time. | Per-call for the long tail; low for the high-volume path. |
| Accuracy control | Prompt and preprocessing edits; the ceiling belongs to the provider’s model. | Directly trainable on your failures; the ceiling belongs to your data quality. | Route each case type to whichever side handles it better. |
| Latency control | Provider-determined, network round trip included. Measure yours. | Yours to engineer — and to keep engineered as load grows. | Fast path where latency binds; API where it does not. |
| Data requirements | None beyond evaluation images. | Hundreds to thousands of labeled images, refreshed as the product drifts. | The custom side needs data only for its narrow slice. |
| Privacy & governance | Images transit a third party — check retention and training-use terms before, not after. | Images stay inside your infrastructure. | Keep the sensitive slice internal; let the rest use the API. |
| Change risk | Provider deprecations, silent model updates, price changes. | Your retraining debt: drift, data refresh, upgrade cycles. | Both risks, smaller each — and each side hedges the other. |
| Team skills required | Backend integration plus prompt and threshold discipline. | Training, serving infrastructure, and evaluation practice. | Both, but staged — the API path buys time to grow the second skill set. |
The four break points
When teams actually move from buy to build
In practice the move is justified by one of four break points — each with a cheap check to run before committing a quarter to the expensive answer.
Unit economics at volume
Per-call pricing that was noise in the prototype dominates the feature’s P&L at production volume. Before building: compute break-even with your own numbers — provider price, 12-month volume, retry and review rates, and the engineering cost you assign to owning a model. No published break-even applies to you.A hard latency floor
The decision moves into a synchronous flow with a budget no hosted round trip can meet. Before building: confirm the budget is real (measured, not asserted) and that caching, batching, or moving the check off the critical path cannot meet it — those cost days, not quarters.Data that cannot leave
Privacy, residency, or contractual terms rule out third-party processing. Before building: check whether a self-hosted open model already covers the task — "custom" and "self-hosted" are different decisions, and the second is often enough.A persistent accuracy gap on your slices
The API keeps failing your decisive cases. Before building: confirm the gap survives better prompts, preprocessing, and thresholds — that check costs an afternoon; the rebuild costs a quarter. A gap that survives is the strongest of the four reasons, because training on your failures is the one lever an API never gives you.
Worked example
The same product at two volumes
Illustrative and fictional — the point is the shape of the reasoning, not the numbers. A vertical SaaS extracts five attributes from uploaded product photos.
- At 20,000 images a month, a prompted VLM handles all five attributes. Per-call cost is a rounding error against engineering time, the schema is enforced in the prompt, and attribute definitions can change by editing text. Building custom here would buy nothing that matters yet.
- At 2,000,000 images a month, one attribute — the one on every upload — dominates spend (break point 1). That attribute is a stable, visually learnable classification, so it moves to a small custom classifier trained on reviewer-corrected labels the team already accumulated. The other four attributes, lower-volume and still evolving, stay on the VLM. The stack is now hybrid, and each side is doing the work it is priced for.
When this changes
Provider economics shift
Open-weight models improve
The task keeps changing
Labeling capacity is the constraint
Limitations
What this matrix cannot settle
- It is directional. Rows flip on your data, your volume, and your team — the matrix tells you where to look, not what you will find.
- It assumes the task is solvable by both paths. If no API covers the task at all, the real comparison is between solution families — start with the model-selection tree.
- It prices the first year. Multi-year ownership costs (team turnover, hardware refresh, provider consolidation) push in both directions and deserve their own line items.
In short
Buy a hosted vision API while it covers your task at modest volume; build a custom model only when a break point arrives — per-image economics at volume, a hard latency floor, data that cannot leave, or slice-level accuracy gaps prompting cannot close. Hybrid stacks that give the high-volume slice to a custom model and the long tail to an API are a normal end state.
Related: how to choose a model · how many images you need · feasibility testing