VISIONROUTE

Guide · Pillar: Choose

Custom model or hosted API? Price the whole stack.

Start with a hosted API or prompted VLM when it covers your task and volume is modest: you pay per call and skip training data entirely. Move to a custom model when one of four break points arrives — unit economics at volume, a hard latency floor, data that cannot leave your infrastructure, or accuracy gaps on your slices that prompting cannot close. Many teams end up hybrid, and that is a result, not a compromise.

Published August 31, 2026 · Pillar: Choose · By the VisionRoute team

Definitions

Hosted vision API

A hosted vision API is a third-party service — a specialist endpoint or a general VLM — that takes your images over the network and returns predictions for a per-call price. You rent the model; the provider owns its weights, roadmap, and deprecation schedule.

Custom-trained model

A custom-trained model is one you fine-tune or train on your own labeled images and serve yourself (or through managed hosting). You own its behavior and its marginal cost — and also its data pipeline, retraining cadence, and serving infrastructure.

Who this guide is for

Teams already running (or prototyping) a hosted approach and wondering whether custom is worth it — and teams about to build custom by default who should probably not. It assumes narrow, still-image decisions inside a software product.

The question, precisely

Build vs buy here is not ideology; it is which stack meets your accuracy bar at acceptable cost, latency, and governance for the next year. The matrix prices both sides — including the parts invoices never show, like retraining debt and deprecation risk.

The cost-and-control matrix

What each path costs, beyond the per-image price

No prices are quoted deliberately: provider pricing changes and break-even depends on your volume and the engineering cost you assign. Fill the numbers in with your own assumptions — dated — before deciding. Definitions are in the glossary.

Directional matrix for still-image decisions in software products. Every row that matters to your decision deserves a measurement, not a citation of this table.
DimensionHosted API / VLMCustom-trained modelHybrid
Time to first resultHours to days: integrate, prompt, threshold.Weeks: collect, label, train, serve, evaluate.Hours for the API path; custom follows where volume concentrates.
Upfront costIntegration effort only.Labeling, training runs, serving setup, and an evaluation harness.Staged — custom costs are paid only for the slice that earns them.
Marginal cost per imageThe per-call price, forever; grows linearly with volume.Low once served; you pay for capacity instead, including its idle time.Per-call for the long tail; low for the high-volume path.
Accuracy controlPrompt and preprocessing edits; the ceiling belongs to the provider’s model.Directly trainable on your failures; the ceiling belongs to your data quality.Route each case type to whichever side handles it better.
Latency controlProvider-determined, network round trip included. Measure yours.Yours to engineer — and to keep engineered as load grows.Fast path where latency binds; API where it does not.
Data requirementsNone beyond evaluation images.Hundreds to thousands of labeled images, refreshed as the product drifts.The custom side needs data only for its narrow slice.
Privacy & governanceImages transit a third party — check retention and training-use terms before, not after.Images stay inside your infrastructure.Keep the sensitive slice internal; let the rest use the API.
Change riskProvider deprecations, silent model updates, price changes.Your retraining debt: drift, data refresh, upgrade cycles.Both risks, smaller each — and each side hedges the other.
Team skills requiredBackend integration plus prompt and threshold discipline.Training, serving infrastructure, and evaluation practice.Both, but staged — the API path buys time to grow the second skill set.

The four break points

When teams actually move from buy to build

In practice the move is justified by one of four break points — each with a cheap check to run before committing a quarter to the expensive answer.

  1. Unit economics at volume

    Per-call pricing that was noise in the prototype dominates the feature’s P&L at production volume. Before building: compute break-even with your own numbers — provider price, 12-month volume, retry and review rates, and the engineering cost you assign to owning a model. No published break-even applies to you.
  2. A hard latency floor

    The decision moves into a synchronous flow with a budget no hosted round trip can meet. Before building: confirm the budget is real (measured, not asserted) and that caching, batching, or moving the check off the critical path cannot meet it — those cost days, not quarters.
  3. Data that cannot leave

    Privacy, residency, or contractual terms rule out third-party processing. Before building: check whether a self-hosted open model already covers the task — "custom" and "self-hosted" are different decisions, and the second is often enough.
  4. A persistent accuracy gap on your slices

    The API keeps failing your decisive cases. Before building: confirm the gap survives better prompts, preprocessing, and thresholds — that check costs an afternoon; the rebuild costs a quarter. A gap that survives is the strongest of the four reasons, because training on your failures is the one lever an API never gives you.

Worked example

The same product at two volumes

Illustrative and fictional — the point is the shape of the reasoning, not the numbers. A vertical SaaS extracts five attributes from uploaded product photos.

  • At 20,000 images a month, a prompted VLM handles all five attributes. Per-call cost is a rounding error against engineering time, the schema is enforced in the prompt, and attribute definitions can change by editing text. Building custom here would buy nothing that matters yet.
  • At 2,000,000 images a month, one attribute — the one on every upload — dominates spend (break point 1). That attribute is a stable, visually learnable classification, so it moves to a small custom classifier trained on reviewer-corrected labels the team already accumulated. The other four attributes, lower-volume and still evolving, stay on the VLM. The stack is now hybrid, and each side is doing the work it is priced for.

When this changes

Provider economics shift

Price cuts and smaller, cheaper hosted models move break point 1 outward; recompute break-even at each volume milestone instead of trusting last year’s decision either way.

Open-weight models improve

A self-hosted open model can dissolve the governance break point without full custom training. Re-check what "custom" is actually buying you beyond self-hosting.

The task keeps changing

Attribute definitions in flux favor the API/VLM side regardless of volume — retraining cycles cannot chase a moving spec as fast as a prompt edit.

Labeling capacity is the constraint

A custom model without enough labeled data underperforms the API it was meant to beat. If you cannot label roughly 500–5,000 images (task-dependent), break point 4 is not actionable yet.

Limitations

What this matrix cannot settle

  • It is directional. Rows flip on your data, your volume, and your team — the matrix tells you where to look, not what you will find.
  • It assumes the task is solvable by both paths. If no API covers the task at all, the real comparison is between solution families — start with the model-selection tree.
  • It prices the first year. Multi-year ownership costs (team turnover, hardware refresh, provider consolidation) push in both directions and deserve their own line items.

In short

Buy a hosted vision API while it covers your task at modest volume; build a custom model only when a break point arrives — per-image economics at volume, a hard latency floor, data that cannot leave, or slice-level accuracy gaps prompting cannot close. Hybrid stacks that give the high-volume slice to a custom model and the long tail to an API are a normal end state.

Related: how to choose a model · how many images you need · feasibility testing