Building the Training and Evaluation Infrastructure for a Production Vision Model

An LLM labeling service on Gemini and Claude, versioned training data, shadow serving and a promotion gate: the infrastructure behind a production vision model.

Teams that want a custom vision model usually start with the model, and the model is the easy part. Whether it's any good in production depends on the infrastructure around it: where the labels come from, how training data is versioned, how a new model sees live traffic without risk, and what decides whether it ships. I see that infrastructure missing all the time.

On a recent engagement, an outdoor camera network wanted to replace the model that classifies its images. I built that infrastructure end to end: an LLM labeling service on Gemini Flash and Claude, a versioned path from approved labels to trained models, an event-driven serving path with a shadow mode, and a promotion gate that's allowed to say HOLD. The model it produced now beats the frontier labs' general-purpose vision models on this task, and the client's team runs the whole loop on their own retrains.

The situation

The network's cameras send images around the clock, and each site also reports readings from its own environmental sensors. A vision model classifies every image on several outputs: the condition of the scene, the condition of the surfaces in view, and whether the lens itself is obstructed. Customers act on alerts built from those classifications.

People could label a thousand or two images a year, far too few to train a new model, and almost none covered the rare conditions that matter most. Promotion meant a person reading a validation report and moving a pointer to the new model.

A labeling service on Gemini Flash and Claude

An LLM drafts every label

The labeling service sends each sampled image to a vision-capable LLM, with a prompt that describes every output and class, and gets back candidate labels and a confidence score. It runs on Gemini Flash-Lite:

• Queue-driven serverless workers with capped concurrency, so big labeling batches drain at a steady rate.
• A first-party cloud provider, not an aggregator, for data governance. The workers reach it through federated identity, with no stored keys.
• Prompts that ship with the function, with the version set in the deploy config. Every label records the model and prompt version that drafted it, so a prompt change never mixes silently with older labels.
• A deduplicating label store, so a retried message doesn't count twice, and an audit row for every image the service skips.

Flash was surprisingly weak at this, even the latest generation. What the model was shown mattered more than the prompt's wording: cropping each image to the region an output is about, rejecting frames too dark to read, and giving it room to reason all helped, and cropping alone took coverage on the hardest output from under one image in ten to more than half. Its drafts still had systematic misses, though, so building the training set took a lot of human labeling and review.

A sampler decides which images get labeled. It uses forecasts and sensor readings to aim at the rare conditions the dataset was missing.

A person approves every label

The LLM's labels are drafts, and a person approves or corrects every one before it trains anything. The model's own confidence score couldn't decide which drafts to check first: an LLM's self-reported confidence is poorly calibrated, and it's often most confident where it's systematically wrong. So the review screen checks each draft against the site's sensors and forecast instead. Drafts the sensors agree with are pre-selected for bulk approval, contradictions go to the top of the queue, and anything rare or ambiguous opens in the full review view. The screen never approves anything on its own.

Claude finds the new model's blind spots

Claude's job is judging. When the old and new models disagree confidently on a live image, Claude looks at it without knowing which model said what and decides which one was right. The images the new model got wrong go back through the labeler and into review, so the next training set covers its blind spots. The client's backend lead built the comparison tool. On one pass I ran with it, most of the new model's mistakes were on night infrared frames, which the training data had barely covered.

A golden set that training never touches

Every decision in this pipeline rests on one asset: a golden set. It's a fixed collection of frames, labeled by people rather than by the LLM, that every model is scored on: the production model, every candidate and every retrain. It was the first thing I built, and I labeled the first version by hand.

If evaluation frames can reach training, your scores measure memory, not the model.

What a golden set buys you

• Scores you can compare. Before, each candidate was scored on its own training run's holdout, so two models' numbers came from two different tests. On one fixed set, a difference in score is a difference in models.
• Ground truth the labeler didn't write. This matters most in an LLM-labeled pipeline. Score models against LLM labels and you're measuring how well they agree with Flash, systematic mistakes included. Human labels are the only thing in the loop that can catch the labeler being wrong.
• A paired comparison. Scoring both models on identical frames is what lets the gate measure the difference between them directly, instead of comparing two noisy scores.
• Honest coverage. Scores are reported per output, per class and per slice, such as day and night frames. A class with fewer than five examples is reported as unmeasured rather than given a number that looks precise and means nothing.
• A versioned record. The set is frozen and versioned, so every score says which version it was measured on, and the production model is re-scored on every new version so the baseline moves with it.

Keeping evaluation frames out of training

A golden set is only worth anything if no model has trained on it, and leakage is easier than it looks. When I audited the first version, about one pick in eight had already leaked into training, through two routes that both look harmless:

• Duplicate label rows. The golden flag lived on each label row, not on the image. When an image picked up a second label, from a re-review or a new prompt version, that row went straight into training.
• Near-identical frames. A camera that takes a picture every few minutes produces frames that are nearly the same. Excluding the exact image does nothing when its neighbor from five minutes earlier is in the training set: the model has effectively seen the answer.

The fix treats the evaluation set as groups, not images. Frames from the same camera within 24 hours form one group. I picked that window by measuring how quickly frames from one camera stop looking alike: a day apart, they're nearly as different as frames from two different cameras. The training query excludes every group with a golden frame in it, keyed on the image and the group, never on a label row, and the build stops if that removes an unusually large share of the data, because that usually means something upstream changed. A verification run after the fix caught the leaked images and the near-duplicates the old filter had missed.

The set started at under a hundred frames and grew to a couple of hundred by autumn, aimed at the conditions the first version couldn't measure. The client's backend lead now publishes it straight from dataset generation, so training data and evaluation data always come from the same split.

From approved labels to a trained model

Training data comes from that one query over the label store: approved labels only, golden groups removed. The result is packed into versioned shards in object storage and trained as containerized jobs on managed GPUs. Every model artifact traces back to the dataset version it was built from, and from there to the labels and prompt versions inside it.

The model answers all of its questions from one shared image backbone, with a separate head for each output. Some outputs pick exactly one class and others can flag several, so each head has its own activation and loss. The site's sensor and forecast readings go in alongside the image. I compared a convolutional backbone with a vision transformer on the same data; the transformer won narrowly.

The export needed its own check. Running the same weights at full precision and through the compressed export, accuracy on one output fell by about three-quarters, so compression stays off by default.

Serving it, and shadowing the new model

Inference is event-driven. An event-bus rule fans every new image out to a queue per model, and an inference function scores the image together with the site's latest sensor readings.

That serving path had a bug training couldn't see. A timezone-naive timestamp key never matched a timezone-aware lookup, so the deployed model got zeros instead of sensor readings. Training was immune because both sides were naive. It was caught before the new model took live traffic.

To see how a candidate behaves on real traffic without risk, I added a shadow mode. Every live image goes to both models, and the candidate writes only to an archive table. A single-writer guard makes sure the live results table has exactly one writer: any configuration value other than the live one fails safe to the archive, and rollback is a flag change. The deploy was verified with zero candidate rows in the live table. The shadow run produced millions of paired live predictions, which showed exactly where the two models disagreed.

A promotion gate that can say no

Before the gate, the validation report couldn't be trusted. It applied a per-label threshold meant for independent labels to outputs that split one probability across several classes, so labels the model got right most of the time scored zero. Each candidate was scored on its own training run's holdout. And it always named a winner. The validator now scores each output exactly as production does, at the production threshold, on the golden set.

Each bootstrap replicate resamples the shared frames once and scores both models on that same sample, so what gets measured is the difference. PROMOTE requires the confidence interval on the overall gain to sit above zero and no output to be distinguishably worse. Anything else is HOLD. Comparing each model's own interval instead would ignore how strongly their errors move together, and on an evaluation set this size it would almost never open.

def verdict(cand, prod, frames, n=1_000, alpha=0.05):
    overall, per_head = [], {h: [] for h in POWERED_HEADS}
    for _ in range(n):
        s = resample(frames)                       # one sample, scored by both models
        deltas = {h: f1(cand, s, h) - f1(prod, s, h) for h in POWERED_HEADS}
        for h, d in deltas.items():
            per_head[h].append(d)
        overall.append(mean(deltas.values()))
    gain_lo, _ = percentile_ci(overall, alpha)
    worse = [h for h, ds in per_head.items() if percentile_ci(ds, alpha)[1] < 0]
    return "PROMOTE" if gain_lo > 0 and not worse else "HOLD"

def verdict(cand, prod, frames, n=1_000, alpha=0.05):
    overall, per_head = [], {h: [] for h in POWERED_HEADS}
    for _ in range(n):
        s = resample(frames)                       # one sample, scored by both models
        deltas = {h: f1(cand, s, h) - f1(prod, s, h) for h in POWERED_HEADS}
        for h, d in deltas.items():
            per_head[h].append(d)
        overall.append(mean(deltas.values()))
    gain_lo, _ = percentile_ci(overall, alpha)
    worse = [h for h, ds in per_head.items() if percentile_ci(ds, alpha)[1] < 0]
    return "PROMOTE" if gain_lo > 0 and not worse else "HOLD"

The gate decides whether a candidate can be promoted, and a person decides whether it is. The client's backend lead built a model-ops screen on the gate's output. Only an admin group can promote, the screen shows the per-output verdicts and flags thin samples, and the model pointer is a versioned configuration value, so every promotion is recorded and rollback is one step.

Results

The new architecture went through the gate in spring and early summer. The October rows are the client's team's own retrains.

The result is a model that beats general-purpose vision on this job. It gets close to 96% of labels right, where the frontier labs' general-purpose vision models average 80 to 85% on the same task. Customers act on these alerts, so that gap is critical: it's the reason a custom model was worth building at all.

The team held one passing candidate because a review of hard cases found it fired far more false positives on some surfaces. They promoted the other, which caught most of a rare condition the previous model had missed, at a small cost on a common one. The move to the new architecture itself was a product decision the team made in late summer, cut over with the same flag the shadow mode used. Every retrain since has gone through the gate, and the team grew the evaluation set to a couple of hundred frames.

What to check in your own AI infrastructure

• Treat LLM labels as drafts. Have a person approve every one, and order the review by independent evidence, not the model's own confidence.
• Keep a golden set no model has seen. Label it by hand, split it by group rather than by image, and check for leaks on every build.
• Version everything between a label and a model. The prompt version on every label, the dataset version on every model, and the live model behind a pointer you can roll back.
• Shadow before you switch. Give the live results exactly one writer, and make every other setting fail safe to an archive.
• Test the serving path, not just the model. Compression and a serve-time feature lookup both broke silently, and neither showed up in training metrics.
• Make HOLD a normal answer. If your comparison can't say "no better", it isn't a gate.

When this is more than you need: when nobody owns review and the golden set can't grow; when the decision is about architecture or product rather than model versions; when you have to ship faster than a person can read a verdict; and when the conditions that matter most haven't happened yet, because you can't label a season the fleet hasn't seen.

Related

• AI systems services: production ML and agent systems, with the infrastructure and evaluation they need.