VISIONROUTE

Guide · Pillar: Prove

Your data, your definition of correct, identical conditions.

To test vision models credibly: assemble 100–300 representative images with recorded expected answers, freeze the scoring rules before any run, execute every candidate on identical inputs with configurations recorded, count refusals and timeouts as failures, and read results by slice rather than average. The CSV template below structures the whole run. A leaderboard rank is evidence about someone else’s task — this method produces evidence about yours.

Published August 31, 2026 · Pillar: Prove · By the VisionRoute team

Definitions

Evaluation vs demo

A demo asks "can it work?" on examples chosen to succeed. An evaluation asks "does it meet the bar?" on examples chosen to be representative — including the ones chosen to make it fail. Same models, opposite selection pressure; only one of them predicts production.

Ground truth

Ground truth is the expected answer for each test input, recorded before testing by whoever owns the definition of correct. If two experts disagree on ten images, the project has a definition problem no model can solve.

Slice

A slice is a named subset of the test set — an input type, source, or difficulty class — scored separately. Averages hide exactly the failures that hurt; slices are where evaluation earns its cost.

Who this is for

Engineers who want to run the comparison themselves. It needs a spreadsheet, an afternoon of scripting, and discipline about the ordering of steps — no evaluation infrastructure required at this scale.

The method

Six steps, in this order

The order is the method: ground truth before runs, scoring rules before results, slices before conclusions. It is the same discipline VisionRoute applies at larger scale.

  1. Build the test set backwards from failure

    Collect 100–300 images that mirror production: common cases, the rare-but-costly cases you fear, true negatives (images where the right answer is "nothing" or "reject"), and quality variation — blur, glare, odd crops. A set assembled from easy examples produces a confident wrong answer.
  2. Record ground truth before any model runs

    Write the expected answer for every image, and resolve disagreements between humans now. Ground truth recorded after seeing model output inherits the model’s mistakes — this ordering is the cheapest integrity guarantee in the whole method.
  3. Freeze the scoring rules

    Decide before the first call: exact match or tolerance? How are partial answers scored? And the failure rules — refusals, timeouts, and malformed output count as failures, not as missing data. Write the rules down; changing them after seeing results is how every approach "wins".
  4. Run every candidate under identical conditions

    Same images, same preprocessing, same number of attempts. Record the exact model or API version, prompt, parameters, and date for each run — providers update models under stable names, and a result without its configuration cannot be reproduced or trusted.
  5. Score everything, and label the failures

    Attach a failure type to every miss: wrong answer, missed object, hallucinated field, refusal, timeout, malformed output. Failure types are where the engineering information lives — two approaches with equal accuracy and different failure types are not equal in production.
  6. Read results by slice, then decide

    Break scores out by the slices you defined: case type, image source, difficulty. A model that wins the average and loses your decisive slice loses. If the difference between candidates is within noise on a small set, the cheaper, simpler stack wins by default; if every candidate fails a decisive slice, the honest verdict is "not enough evidence" or a reformulated task.

The template

One row per example, per candidate

The template is long-format: each row is one candidate’s result on one example, which makes slice pivots and cross-run joins trivial in any spreadsheet tool.

Two example rows ship inside the file, clearly marked EXAMPLE — delete them before recording real results.
ColumnWhat goes in it
example_idStable ID for the test image, so results join across runs.
image_fileFile name or storage path of the image.
sliceThe slice this example belongs to — e.g. common, hard_glare, negative.
is_negativeTRUE when the correct answer is "nothing" / "reject". Negatives keep false positives honest.
expected_outputGround truth, recorded before testing (step 2).
hard_case_reasonWhy this example is hard, when it is — occlusion, lookalike, rare class.
candidateWhich model, API, or stack produced this row.
candidate_versionExact model/API version or build. No version, no comparison.
run_configPrompt or configuration reference used for this run.
run_dateDate of the run — provider behavior changes over time.
raw_outputWhat the candidate actually returned, verbatim.
scored_correctTRUE/FALSE under the frozen scoring rules (step 3).
failure_typewrong_answer, missed, hallucinated, refused, timeout, malformed — or empty when correct.
latency_msMeasured wall-clock latency for this call, from your infrastructure.
notesAnything a future reader needs to believe this row.
Vision-model evaluation templateThe fifteen columns above with two marked example rows — imports into any spreadsheet, database, or notebook.Download ↓

When this changes

The task is subjective

When correct is a judgment ("is this image appealing?"), replace exact-match scoring with a written rubric and multiple raters, and measure agreement between raters before measuring any model. Low human agreement bounds every model score.

You have fewer than ~50 images

Run the method anyway, but treat results as directional: rank candidates, do not certify them. Small sets separate approaches that differ a lot; they cannot resolve close calls or rare slices.

Output is continuous or structured

Counts, coordinates, and multi-field extractions need per-field tolerances defined in step 3 — "how wrong is still right" becomes the scoring rule, and partial credit must be decided before results exist.

Limitations

Where the DIY method stops

  • At 100–300 images, it reliably separates approaches that differ meaningfully. Close calls, rare slices, and statistical confidence need more data and more careful analysis — do not claim a decisive result the set cannot support.
  • It is a point-in-time result. Models and providers change; the test set you built here should become a regression suite you rerun on every stack change.
  • It measures the model, not the stack around it — thresholds, review paths, and integration behavior still need the production checklist before shipping.

In short

Testing vision models on your own data takes six ordered steps: build a representative 100–300 image set backwards from failure, record ground truth first, freeze scoring and failure rules, run identical conditions with versions recorded, label every failure by type, and read results by slice. Refusals and timeouts count as failures, and close calls default to the cheaper, simpler stack.

Related: comparing APIs fairly · how many images you need