Case Study — Vision AI & Edge
94% in the lab.
61% on the line.
Illustrative scenario. This is a pattern-level write-up of a recurring failure mode and its fix — not a write-up of one named client project. Stack, numbers, and timeline are representative of this class of engagement, not a specific client's exact figures.
Pattern: Manufacturing quality inspection
Illustrative timeline: ~7 weeks
Related: Vision AI & Edge Deployment
By Rajeish Mata · Reviewed by the Dwayo AI engineering team · Published October 12, 2026 · Updated October 12, 2026
The problem
94% in the lab. 61% on the line.
A team had trained a computer vision model to detect defects on a production line, validated against a held-out test set at 94% accuracy. When the model moved to the actual camera and lighting setup on the floor, accuracy dropped to roughly 61% — good enough to erode trust in the system, not good enough to rely on. The gap wasn't a flaw in the model architecture. It was a mismatch between training data and deployment conditions. The training set was captured under studio-quality lighting that the actual production floor never provided.
This lab-to-field gap is the same class of problem we see across manufacturing and agritech deployments generally — not specific to this engagement, but common enough in that sector that we built field validation into the default process rather than treating it as an edge case.
Evaluation method
Validated where it runs, not where it was trained.
The acceptance criterion for this class of fix is specific: accuracy has to be measured on a held-out set captured under real field conditions — the same camera, lighting, and line speed the model will actually run against — not the lab test set the model was originally validated on. A model that looks good only on studio images has not yet been evaluated for the job it's being deployed to do.
Security & governance
Hardware validation as a release gate.
For edge-deployed vision systems, a model update isn't safe to ship just because it scored well in a training run — it has to be validated on the specific edge hardware and camera setup it will run on, since inference behavior can differ from a training environment's GPU. We treat a hardware-specific validation pass as part of the release process, not an optional check.