Guide · Pillar: Prove
Same inputs, same constraints, same failure rules.
A credible image-recognition API comparison fixes everything except the API: identical images, scoring rules frozen before the first call, refusals and timeouts counted as failures, latency measured from your infrastructure, and cost computed under your usage assumptions rather than list prices. Run at least 100 representative images including your hard cases, score by slice, and record the date and model versions — providers change under you. The scorecard below holds all of it.
Published August 31, 2026 · Pillar: Prove · By the VisionRoute team
Definitions
Image-recognition API
Failure rules
Equivalent usage assumptions
Who this is for
The protocol
Six rules that keep the comparison honest
This is the same discipline as testing models on your own data, specialized for the vendor decision — and the standard VisionRoute holds its own comparisons to.
Fix the task and map every API to one output schema
Each API speaks its own dialect — different label taxonomies, score scales, and response shapes. Define one canonical output for your task and write the mapping from each API into it before scoring anything, or you will compare formats instead of quality.Run identical inputs
Same images, same resolutions, same preprocessing, same retry policy, run in the same period. If one API sees cleaned images or extra attempts, the comparison is already over.Freeze scoring and failure rules first
Decide before the first call how answers are scored and what counts as failure. Refusals, timeouts, and malformed responses are failures — the moment they become "excluded rows", the least reliable API starts winning.Measure latency yourself
p50 and p95 from your infrastructure and region, under your concurrency — not the number on the pricing page. Network path and payload size routinely dominate model time.Compute cost under equivalent usage assumptions
Pick your assumptions — monthly volume, image sizes, retry rate, expected human-review rate — and price every API under the same ones. List prices with different units ("per image" vs "per 1,000 units" vs tiered) are not comparable until you do.Score by slice, then date and version the verdict
Read accuracy per slice including negatives, weight the dimensions by weights you wrote down before running, and record the run date and each API’s model version. The result is true for that date and those versions — treat it as perishable.
The scorecard
Seven sections, one page per vendor decision
| Scorecard section | What it captures | The trap it prevents |
|---|---|---|
| Setup & versions | Task definition, run date, model/API version, and configuration per candidate. | Comparing one vendor’s spring model against another’s autumn model and calling it a finding. |
| Quality by slice | Accuracy per slice — common cases, hard cases, negatives — under the frozen rules. | One average hiding a failure on exactly the slice the product depends on. |
| Failure accounting | Refusal, timeout, and malformed-response rates, counted as failures. | Averaging reliability problems away as "missing data". |
| Latency | p50/p95 measured from your infrastructure, per API. | Quoting the provider’s published latency as if your users lived in their data center. |
| Cost under your assumptions | Per-1,000-image cost at your volume, image size, retry and review rates — same assumptions for all. | Comparing list prices denominated in different units. |
| Governance | Retention, training-use, and residency terms per provider. | Discovering the data policy after the integration ships. |
| Weighted verdict | Dimension weights written down before the run, and the resulting ranking. | Letting whichever candidate won pick the scoring after the fact. |
When this changes
No API covers the task
Volume is small and stakes are low
The result is a near-tie
Limitations
What a scorecard cannot fix
- It is a point-in-time result. Providers ship new models and reprice without asking; the recorded date and versions are what make the verdict honest — and re-runnable.
- It cannot rescue an unrepresentative test set. Garbage in, confident garbage out — set size and composition are covered in how many images you need.
- It compares hosted candidates only. Whether a custom model beats the winning API is a separate decision with its own break points.
In short
To compare image-recognition APIs fairly: map every API to one output schema, run identical images with frozen scoring, count refusals and timeouts as failures, measure p50/p95 latency from your own infrastructure, price all candidates under the same usage assumptions, and score by slice with weights chosen before the run — then date and version the verdict, because it is perishable.
Related: the DIY evaluation method · custom model vs hosted API