Creating a pre-launch AI pipeline test protocol
Identified a production pipeline discrepancy on launch day that was silently dropping prompt engineering improvements, then created a repeatable pre-launch testing process that became a permanent part of the ship checklist.
The challenge
On the day of the alpha launch for a Meta Horizon creator feature — a 3D retexturing AI tool for designing and selling avatar clothing — I noticed that outputs in the live product didn't match what our internal testing environment had been producing.
The implication was serious. Ship readiness had been evaluated entirely against the internal environment, which meant the quality improvements from weeks of prompt engineering and model tuning had been signed off on a version of the system that wasn't actually running in production. And the failure was silent: nothing had erred, no alert had fired, no automated check had caught it. The pipeline had simply never picked up the changes — and without a side-by-side comparison, there was nothing to indicate anything was wrong.

The solution
I created a repeatable pre-launch testing protocol that uses matched prompts and seeds to directly compare outputs between the internal testing environment and the production model. The principle is simple: the same prompt and the same seed should produce identical results across environments. When they don't, the divergence is immediately visible — and documentable.
That single constraint turned an invisible failure into a catchable one. The protocol was then formalized as a standing item on the pre-launch checklist and is run every time a model upgrade ships.
Launch day output comparison — same prompt, same seed
Internal testing environment - prompt engineering applied

Production (creation tool) - Missing vibrancy & accuracy fixes

Test prompt used
"a garment with an astronaut aesthetic made from a combination of durable, high-tech materials such as metallic-coated nylon and insulated polyester. the surface features a repeating pattern of galaxy-inspired motifs in shades of deep blue and purple, with accents of bright orange and yellow. the design includes functional details like reinforced seams, utility pockets, and adjustable cuffs, all crafted to evoke the sense of adventure and exploration associated with space travel. the overall texture is rugged and utilitarian, with a mix of smooth and textured elements that convey a sense of durability and functionality."
My role
As the senior UX lead and the team's subject-matter expert on model quality, I caught the discrepancy by noticing subtle but meaningful differences in launch-day outputs — differences no automated check had flagged. I documented the divergence side by side using matched prompts and seeds, which gave research scientists and product engineers the precise evidence they needed to trace the issue to its root cause: the production pipeline hadn't ingested the latest prompt and model changes.
Once the alpha launched with the issue resolved, I established the same comparison test as a permanent pre-launch requirement — turning a one-time catch into a standing safeguard against the same failure mode recurring on future model upgrades.
Results & impact
The issue was caught and resolved before it could meaningfully affect creator output quality during the alpha. The alpha went on to generate $83.3K in revenue from 38.5K clothing items sold in its first month — across just 20 creators — validating both the product and the quality bar it launched at.
The more lasting outcome was the protocol itself. The seed-and-prompt comparison test became a repeatable pre-launch standard, giving the team a reliable way to verify that prompt engineering and model tuning changes are fully reflected in production before anything ships. A failure that nearly went out silently became the reason the team can now trust what it ships.