Guide · Pillar: Ship
What a convincing demo still has not proved.
A demo proves the happy path exists. “Production-ready” is a property of the whole stack: an enforced output contract, thresholds with an abstention path, a human-review loop, versioned models and prompts, latency and cost measured at production volume, drift monitoring, a regression suite, and a rollback plan. The checklist below covers those layers in eighteen checks; the release gate at the end is deliberately pass/fail.
Published August 31, 2026 · Pillar: Ship · By the VisionRoute team
Definitions
"Production-ready", operationally
Confidence threshold
Regression suite
Who this is for
The checklist
Eighteen checks across six layers of the stack
Output contract & thresholds (1–3)
1. Every response is validated against a schema.
Malformed or off-schema output takes a defined failure path — it never crashes the flow and never silently passes as a prediction.
2. Confidence thresholds come from your evaluation data.
Set per failure direction, from the measured tradeoff on representative images — not from a library default or a demo that felt right.
3. Below-threshold behavior is specified and implemented.
Abstain, queue for review, or apply a safe default — chosen per decision, in writing.
Review & abstention (4–6)
4. The human review queue has an owner and an SLA.
A queue nobody drains is a silent outage. Someone owns throughput, and the product defines what happens while an item waits.
5. Reviewer decisions are captured as labeled data.
Every override is a free training and regression example. Losing them is paying for labels and throwing them away.
6. The abstention rate is budgeted and monitored.
Decide the acceptable rate up front. A model that quietly routes 30% of traffic to humans has changed your unit economics without telling you.
Versioning & change (7–9)
7. Every decision records its full configuration.
Model or API version, prompt, thresholds, preprocessing — enough to reproduce any past decision when a customer asks why.
8. No change ships without a regression run.
Model swap, prompt edit, threshold tweak, provider update — each runs the evaluation suite first, and the diff is read by slice.
9. Provider change policy is explicit.
Pin model versions where the provider allows it; where it does not, monitor for behavior shifts you did not initiate.
Monitoring & drift (10–12)
10. Input drift has signals.
Source mix, image quality, class balance — the distribution you validated against is the one assumption production erodes first.
11. Outcome quality is tracked per slice in production.
Direct labels where you have review coverage; proxies like overturn rate and customer disputes where you do not.
12. Alerts have thresholds and a named receiver.
A dashboard nobody is paged on is documentation, not monitoring.
Economics & latency (13–15)
13. Latency is measured at production volume.
p50 and p95, on your serving path, from your region, under real concurrency — not the provider’s published figure.
14. Per-decision cost is computed end to end.
Including retries, preprocessing, storage, and the human-review minutes each abstention buys.
15. Economics are re-checked at volume milestones.
The stack that made sense at 5,000 images a day can be the wrong stack at 50,000 — put the recheck on the calendar.
Rollback & incident (16–18)
16. The previous version is deployable quickly.
Model, prompt, and thresholds roll back together as one unit — a model rollback under new thresholds is a third, untested configuration.
17. A kill switch exists.
One action routes all traffic to human review or a safe default. Rehearsed, not theoretical.
18. The incident playbook names the first three checks.
Did the input distribution change? Did the provider change? Did we change a threshold? Most vision incidents are one of these three.
The release gate
Eight statements. All true, or not yet.
The checklist above is for building; this gate is for the launch meeting. Read each statement aloud; ship when all eight are true of the exact build in front of you.
Acceptance criteria are met on the current regression suite, by slice, on the exact build being shipped.
Every component version in the decision path — model, prompt, thresholds, preprocessing — is recorded and reproducible.
The low-confidence path has been exercised end to end in staging, including the human queue and its notifications.
A named person owns each of: the review queue, drift alerts, and regression reruns.
p95 latency and per-decision cost at expected volume are measured on your serving path, not projected.
Rollback to the previous full configuration has been rehearsed at least once.
Every failure mode with a named business cost has monitoring that would actually catch it.
The team can answer "when do we rerun the suite?" with both a calendar interval and a list of trigger events.
When this changes
Low-stakes internal tools
Regulated and safety-relevant domains
Batch instead of real-time
Limitations
What passing the gate does not mean
- It does not certify future behavior. Models, providers, and your own input distribution drift; the gate is re-earned at every material change, which is what the regression suite is for.
- It cannot price your failures for you. The checklist assumes accuracy bars and failure costs already exist — those come from the feasibility checklist and your domain owner.
- It is scoped to still-image decisions in software products. Video, robotics, and edge deployments add layers this page does not cover.
In short
“Production-ready computer vision” is a property of the stack, not the model: enforced output contract, evaluation-derived thresholds with an abstention path, an owned review queue, reproducible versions, a regression suite run on every change, drift monitoring with a named receiver, measured latency and cost at volume, and a rehearsed rollback. Eighteen checks build it; an eight-statement gate decides the launch.
Related: build the evaluation suite · example free report