Back to Case Studies

Creating a pre-launch AI pipeline test protocol

Identified a production pipeline discrepancy on launch day that was silently dropping prompt engineering improvements, then created a repeatable pre-launch testing process that became a permanent part of the ship checklist.

The challenge

On the day of the alpha launch for a Meta Horizon creator feature — a 3D retexturing AI tool for designing and selling avatar clothing — I noticed that outputs in the live product didn't match what our internal testing environment had been producing.

The implication was serious. Ship readiness had been evaluated entirely against the internal environment, which meant the quality improvements from weeks of prompt engineering and model tuning had been signed off on a version of the system that wasn't actually running in production. And the failure was silent: nothing had erred, no alert had fired, no automated check had caught it. The pipeline had simply never picked up the changes — and without a side-by-side comparison, there was nothing to indicate anything was wrong.

Steampunk aesthetic design showing system architecture before and after

The solution

I created a repeatable pre-launch testing protocol that uses matched prompts and seeds to directly compare outputs between the internal testing environment and the production model. The principle is simple: the same prompt and the same seed should produce identical results across environments. When they don't, the divergence is immediately visible — and documentable.

That single constraint turned an invisible failure into a catchable one. The protocol was then formalized as a standing item on the pre-launch checklist and is run every time a model upgrade ships.

Launch day output comparison — same prompt, same seed

Internal testing environment - prompt engineering applied

Astronaut-themed vest with vibrant blue and purple galaxy pattern with orange accents

Production (creation tool) - Missing vibrancy & accuracy fixes

Astronaut-themed vest with muted colors and less vibrant galaxy pattern

Test prompt used

"a garment with an astronaut aesthetic made from a combination of durable, high-tech materials such as metallic-coated nylon and insulated polyester. the surface features a repeating pattern of galaxy-inspired motifs in shades of deep blue and purple, with accents of bright orange and yellow. the design includes functional details like reinforced seams, utility pockets, and adjustable cuffs, all crafted to evoke the sense of adventure and exploration associated with space travel. the overall texture is rugged and utilitarian, with a mix of smooth and textured elements that convey a sense of durability and functionality."


My role

As the senior UX lead and the team's subject-matter expert on model quality, I caught the discrepancy by noticing subtle but meaningful differences in launch-day outputs — differences no automated check had flagged. I documented the divergence side by side using matched prompts and seeds, which gave research scientists and product engineers the precise evidence they needed to trace the issue to its root cause: the production pipeline hadn't ingested the latest prompt and model changes.

Once the alpha launched with the issue resolved, I established the same comparison test as a permanent pre-launch requirement — turning a one-time catch into a standing safeguard against the same failure mode recurring on future model upgrades.


Results & impact

The issue was caught and resolved before it could meaningfully affect creator output quality during the alpha. The alpha went on to generate $83.3K in revenue from 38.5K clothing items sold in its first month — across just 20 creators — validating both the product and the quality bar it launched at.

The more lasting outcome was the protocol itself. The seed-and-prompt comparison test became a repeatable pre-launch standard, giving the team a reliable way to verify that prompt engineering and model tuning changes are fully reflected in production before anything ships. A failure that nearly went out silently became the reason the team can now trust what it ships.

$83.3K
revenue generated in the first month of alpha with 20 creators
38.5K
clothing items sold during the alpha test period