Guide · Pillar: Prove
Your data, your definition of correct, identical conditions.
To test vision models credibly: assemble 100–300 representative images with recorded expected answers, freeze the scoring rules before any run, execute every candidate on identical inputs with configurations recorded, count refusals and timeouts as failures, and read results by slice rather than average. The CSV template below structures the whole run. A leaderboard rank is evidence about someone else’s task — this method produces evidence about yours.
Published August 31, 2026 · Pillar: Prove · By the VisionRoute team
Definitions
Evaluation vs demo
Ground truth
Slice
Who this is for
The method
Six steps, in this order
The order is the method: ground truth before runs, scoring rules before results, slices before conclusions. It is the same discipline VisionRoute applies at larger scale.
Build the test set backwards from failure
Collect 100–300 images that mirror production: common cases, the rare-but-costly cases you fear, true negatives (images where the right answer is "nothing" or "reject"), and quality variation — blur, glare, odd crops. A set assembled from easy examples produces a confident wrong answer.Record ground truth before any model runs
Write the expected answer for every image, and resolve disagreements between humans now. Ground truth recorded after seeing model output inherits the model’s mistakes — this ordering is the cheapest integrity guarantee in the whole method.Freeze the scoring rules
Decide before the first call: exact match or tolerance? How are partial answers scored? And the failure rules — refusals, timeouts, and malformed output count as failures, not as missing data. Write the rules down; changing them after seeing results is how every approach "wins".Run every candidate under identical conditions
Same images, same preprocessing, same number of attempts. Record the exact model or API version, prompt, parameters, and date for each run — providers update models under stable names, and a result without its configuration cannot be reproduced or trusted.Score everything, and label the failures
Attach a failure type to every miss: wrong answer, missed object, hallucinated field, refusal, timeout, malformed output. Failure types are where the engineering information lives — two approaches with equal accuracy and different failure types are not equal in production.Read results by slice, then decide
Break scores out by the slices you defined: case type, image source, difficulty. A model that wins the average and loses your decisive slice loses. If the difference between candidates is within noise on a small set, the cheaper, simpler stack wins by default; if every candidate fails a decisive slice, the honest verdict is "not enough evidence" or a reformulated task.
The template
One row per example, per candidate
The template is long-format: each row is one candidate’s result on one example, which makes slice pivots and cross-run joins trivial in any spreadsheet tool.
| Column | What goes in it |
|---|---|
| example_id | Stable ID for the test image, so results join across runs. |
| image_file | File name or storage path of the image. |
| slice | The slice this example belongs to — e.g. common, hard_glare, negative. |
| is_negative | TRUE when the correct answer is "nothing" / "reject". Negatives keep false positives honest. |
| expected_output | Ground truth, recorded before testing (step 2). |
| hard_case_reason | Why this example is hard, when it is — occlusion, lookalike, rare class. |
| candidate | Which model, API, or stack produced this row. |
| candidate_version | Exact model/API version or build. No version, no comparison. |
| run_config | Prompt or configuration reference used for this run. |
| run_date | Date of the run — provider behavior changes over time. |
| raw_output | What the candidate actually returned, verbatim. |
| scored_correct | TRUE/FALSE under the frozen scoring rules (step 3). |
| failure_type | wrong_answer, missed, hallucinated, refused, timeout, malformed — or empty when correct. |
| latency_ms | Measured wall-clock latency for this call, from your infrastructure. |
| notes | Anything a future reader needs to believe this row. |
When this changes
The task is subjective
You have fewer than ~50 images
Output is continuous or structured
Limitations
Where the DIY method stops
- At 100–300 images, it reliably separates approaches that differ meaningfully. Close calls, rare slices, and statistical confidence need more data and more careful analysis — do not claim a decisive result the set cannot support.
- It is a point-in-time result. Models and providers change; the test set you built here should become a regression suite you rerun on every stack change.
- It measures the model, not the stack around it — thresholds, review paths, and integration behavior still need the production checklist before shipping.
In short
Testing vision models on your own data takes six ordered steps: build a representative 100–300 image set backwards from failure, record ground truth first, freeze scoring and failure rules, run identical conditions with versions recorded, label every failure by type, and read results by slice. Refusals and timeouts count as failures, and close calls default to the cheaper, simpler stack.
Related: comparing APIs fairly · how many images you need