VISIONROUTE

Guide · Pillar: Prove

Same inputs, same constraints, same failure rules.

A credible image-recognition API comparison fixes everything except the API: identical images, scoring rules frozen before the first call, refusals and timeouts counted as failures, latency measured from your infrastructure, and cost computed under your usage assumptions rather than list prices. Run at least 100 representative images including your hard cases, score by slice, and record the date and model versions — providers change under you. The scorecard below holds all of it.

Published August 31, 2026 · Pillar: Prove · By the VisionRoute team

Definitions

Image-recognition API

A hosted service that takes images and returns predictions — labels, boxes, text, fields, or judgments — for a per-call price. Specialist endpoints and general VLM APIs both count here; the comparison method does not care which family a candidate belongs to.

Failure rules

The written policy for what counts as a failure: wrong answer, refusal, timeout, malformed or off-schema output. Declared before testing and applied identically to every candidate — reliability is part of quality, not a footnote.

Equivalent usage assumptions

One set of numbers — monthly volume, image sizes, retry rate, review rate — used to price every candidate. Costs become comparable only when every API is priced against the same imagined month of production.

Who this is for

Teams that have already chosen the hosted-API path and now face three or four plausible vendors. If the open question is still which solution family fits, start with the model-selection guide instead.

The protocol

Six rules that keep the comparison honest

This is the same discipline as testing models on your own data, specialized for the vendor decision — and the standard VisionRoute holds its own comparisons to.

  1. Fix the task and map every API to one output schema

    Each API speaks its own dialect — different label taxonomies, score scales, and response shapes. Define one canonical output for your task and write the mapping from each API into it before scoring anything, or you will compare formats instead of quality.
  2. Run identical inputs

    Same images, same resolutions, same preprocessing, same retry policy, run in the same period. If one API sees cleaned images or extra attempts, the comparison is already over.
  3. Freeze scoring and failure rules first

    Decide before the first call how answers are scored and what counts as failure. Refusals, timeouts, and malformed responses are failures — the moment they become "excluded rows", the least reliable API starts winning.
  4. Measure latency yourself

    p50 and p95 from your infrastructure and region, under your concurrency — not the number on the pricing page. Network path and payload size routinely dominate model time.
  5. Compute cost under equivalent usage assumptions

    Pick your assumptions — monthly volume, image sizes, retry rate, expected human-review rate — and price every API under the same ones. List prices with different units ("per image" vs "per 1,000 units" vs tiered) are not comparable until you do.
  6. Score by slice, then date and version the verdict

    Read accuracy per slice including negatives, weight the dimensions by weights you wrote down before running, and record the run date and each API’s model version. The result is true for that date and those versions — treat it as perishable.

The scorecard

Seven sections, one page per vendor decision

The CSV ships with three candidate columns and a marked instructions row — add columns for more candidates, delete the instructions row before use.
Scorecard sectionWhat it capturesThe trap it prevents
Setup & versionsTask definition, run date, model/API version, and configuration per candidate.Comparing one vendor’s spring model against another’s autumn model and calling it a finding.
Quality by sliceAccuracy per slice — common cases, hard cases, negatives — under the frozen rules.One average hiding a failure on exactly the slice the product depends on.
Failure accountingRefusal, timeout, and malformed-response rates, counted as failures.Averaging reliability problems away as "missing data".
Latencyp50/p95 measured from your infrastructure, per API.Quoting the provider’s published latency as if your users lived in their data center.
Cost under your assumptionsPer-1,000-image cost at your volume, image size, retry and review rates — same assumptions for all.Comparing list prices denominated in different units.
GovernanceRetention, training-use, and residency terms per provider.Discovering the data policy after the integration ships.
Weighted verdictDimension weights written down before the run, and the resulting ranking.Letting whichever candidate won pick the scoring after the fact.
Image-recognition API comparison scorecardAll seven sections as spreadsheet rows — including the usage-assumption block and the pre-committed weights — ready for your candidates.Download ↓

When this changes

No API covers the task

If every candidate fails the decisive slice, the verdict is not "pick the least bad" — it is that the hosted-API path itself is unproven for this task. Compare solution families instead of vendors.

Volume is small and stakes are low

At a few hundred images a month, integration ease and governance terms can legitimately outweigh small accuracy differences. Run the protocol anyway — but weight the dimensions honestly for your situation.

The result is a near-tie

Within noise on your set, the cheapest, simplest candidate wins by default — and the scorecard’s governance and change-risk rows become the tie-breakers.

Limitations

What a scorecard cannot fix

  • It is a point-in-time result. Providers ship new models and reprice without asking; the recorded date and versions are what make the verdict honest — and re-runnable.
  • It cannot rescue an unrepresentative test set. Garbage in, confident garbage out — set size and composition are covered in how many images you need.
  • It compares hosted candidates only. Whether a custom model beats the winning API is a separate decision with its own break points.

In short

To compare image-recognition APIs fairly: map every API to one output schema, run identical images with frozen scoring, count refusals and timeouts as failures, measure p50/p95 latency from your own infrastructure, price all candidates under the same usage assumptions, and score by slice with weights chosen before the run — then date and version the verdict, because it is perishable.

Related: the DIY evaluation method · custom model vs hosted API