Case Study — Vision AI & Edge

94% in the lab.
61% on the line.

Illustrative scenario. This is a pattern-level write-up of a recurring failure mode and its fix — not a write-up of one named client project. Stack, numbers, and timeline are representative of this class of engagement, not a specific client's exact figures.
Pattern: Manufacturing quality inspection Illustrative timeline: ~7 weeks Related: Vision AI & Edge Deployment

94% in the lab. 61% on the line.

A team had trained a computer vision model to detect defects on a production line, validated against a held-out test set at 94% accuracy. When the model moved to the actual camera and lighting setup on the floor, accuracy dropped to roughly 61% — good enough to erode trust in the system, not good enough to rely on. The gap wasn't a flaw in the model architecture. It was a mismatch between training data and deployment conditions. The training set was captured under studio-quality lighting that the actual production floor never provided.

This lab-to-field gap is the same class of problem we see across manufacturing and agritech deployments generally — not specific to this engagement, but common enough in that sector that we built field validation into the default process rather than treating it as an edge case.

What it was built on.

A representative setup for this kind of inspection use case — illustrative of the pattern, not a specific project's exact hardware:

Custom-trained CV model Edge inference device on the line Studio-quality training dataset No field data in training No pre-deployment field validation

The original training data came entirely from a controlled studio capture setup. The production floor's actual camera angle, lighting variance across shifts, and lens characteristics were never represented in anything the model had seen before deployment.

Where field data enters the loop.

Field capture (real line)
→
Augmented training set
→
Edge inference
→
Field validation
→
Low-confidence feedback loop

Train on what the camera will actually see.

The fix wasn't a bigger model. It was closing the gap between training conditions and deployment conditions:

  • Collected real field data from the actual production camera and lighting setup, across different shifts and times of day, instead of relying solely on the clean studio dataset.
  • Augmented training data with synthetic lighting variance, motion blur, and lens distortion matched to the real hardware, to extend coverage beyond what field capture alone provided.
  • Pre-deployment field validation — accuracy measured on the actual line before rollout, not assumed from lab results.
  • Low-confidence feedback loop — cases the model flagged as uncertain were routed for human review and fed back into the next training pass.

Validated where it runs, not where it was trained.

The acceptance criterion for this class of fix is specific: accuracy has to be measured on a held-out set captured under real field conditions — the same camera, lighting, and line speed the model will actually run against — not the lab test set the model was originally validated on. A model that looks good only on studio images has not yet been evaluated for the job it's being deployed to do.

What changed once training matched deployment.

This is the pattern of improvement we typically see once field-matched training and validation go in — illustrative, not a specific client's measured figures.

Before
Accuracy dropped sharply from test set to the production line
After
Validation accuracy measured on real field conditions, not studio images
Ongoing
Low-confidence cases flagged and fed back into training

Hardware validation as a release gate.

For edge-deployed vision systems, a model update isn't safe to ship just because it scored well in a training run — it has to be validated on the specific edge hardware and camera setup it will run on, since inference behavior can differ from a training environment's GPU. We treat a hardware-specific validation pass as part of the release process, not an optional check.

What a client walks away with.

For this class of engagement, the deliverables are the field-validated model, the augmented training pipeline, and the low-confidence feedback loop wired into the client's own retraining cadence — covered further under machine learning development and MLOps drift monitoring.

If this looks familiar.

This is a pattern-level breakdown of what our vision AI and edge deployment work is built to prevent. If a model's production accuracy is quietly worse than its test-set accuracy, it's almost always a training-versus-deployment mismatch — not a model problem. See also machine learning development for the broader validation methodology, and our AI development services for how engagements like this are scoped.

Get started

Tell us where
your model actually runs.

Book a 30-minute call — we'll tell you honestly whether your validation set matches your deployment conditions.