Vanessa Casavant
Back to case studies
Quality hillclimbing

Optimizing an agentic video creation system

Our agentic video creation system beat creator-made reels in a public Facebook test, but only 3% of the videos it generated were good enough to use. I gave a fast-growing team a shared framework to diagnose quality issues, then directed a 1-week sprint that raised the video pass rate to 31%.

Overview

At a glance

My role
Sprint lead and model designer. I set priorities, divided the work, and owned agent persona design; instructions for how agents write prompts and the context they hand off; input pipeline cleanup; and visual style guardrails.
Partners
  • 2 engineers new to the project who integrated game context, built a new quality inspector agent I proposed, and partnered with me on the benchmarking test plan
  • 2 game QA testers on input cleanup, following my guidelines
  • Contract team of human raters scoring videos against my evaluation criteria
Timeline
1 week
Models
Claude, Nano Banana, Veo, and ElevenLabs
System
A multimodal 8-agent pipeline that turns mobile gaming assets like images, gameplay clips, and descriptions into short-form social video.

Headline metrics

3% → 31%increase of videos passing the quality bar, in one week
864videos scored across 6 games to validate the gains
15%fewer visual hallucinations than before the sprint

Our multimodal agentic system turned a mobile game’s images, gameplay clips, and descriptions into short-form videos for social media promotion. Its generated videos beat creator-made reels in a public Facebook test, but a quality problem undercut the win. Using the scoring system I built to rate visual quality, audio quality, and game accuracy, I found only 3% of its videos were good enough to ship, taking about 35 generations to produce one testable video. This failure rate drove high token burn.

Fixing this issue fell to a fast-moving team. The test win brought wave after wave of new people from Horizon, Facebook, and Instagram onto the project. Around the same time, the research scientist who owned the system’s architecture rolled off to other work. Most newcomers didn’t know how agentic systems worked in general, much less this one, so they debugged it ad hoc. As the only person left who knew the whole system end to end, I needed to give everyone a shared way to understand how it worked and where it failed before we could raise the pass rate.

01

Four levers anyone could pull

I mapped everything that could improve the system into 4 levers, each one something you can pull on its own.

Naming the levers turned “the output is bad” into a diagnosis tool. We could trace every failure and needed fix to a lever. It also clarified what we steer by hand with templates and what the agents decide on their own so the system can scale across any game genre. PMs, engineers, data scientists, and designers from three orgs finally had one way to reason about the system together. It’s the framework that made the sprint focused instead of frantic.

Diagnostic framework

4 levers for quality

What the levers control

Inputs

Game info and source assets the system creates from

Inputs

Game contextVisual assets

Templates

Unique system prompts for scene planning and story

Templates

HookStory arcScene progression

Agents

Instructions, personas, and context passed between agents

Agent behavior

Creative decisionsHandoffsGuardrails

Models

Underlying models for agents and generators

Models

InspectionSamplingGeneration
02

A hypothesis about where the failures lived

My hypothesis was that the biggest failures came from inputs and agent behavior, not the templates of unique scene structures. So that’s where I aimed the sprint, directing two engineers who were new to the project across 3 of the 4 levers.

  • Inputs: We scrubbed the asset folders to remove bad reference images before they entered the pipeline.
  • Agent behavior: We threaded game context through every creative agent, gave each agent a detailed persona, cached input-asset descriptions, overhauled the prompt-writing guidance, raised sampling rates, and added avatar-style guardrails.
  • Models: We added an additional quality inspection agent further upstream to catch bad first-frame generated reference images for video generation, because a single bad reference frame could poison an entire scene.

Failure types before the sprint

Visual hallucinations
Game inaccuracies
Incoherent story
Avatar style issues

Illustrative distribution of failure types observed before the sprint based on the eval criteria I wrote for visual quality, audio quality, and game accuracy.

A failed “before” video with many of the quality issues we needed to solve.

03

Testing against the hardest cases

To validate the gains, we generated 864 videos across 6 Horizon games and had reviewers score them against evaluation criteria I wrote to measure visual quality, audio quality, and game accuracy. We deliberately sampled 3 games with the highest pass rates and 3 with the lowest, so the fixes faced easy and hard cases alike, then compared failure types before and after.

Scoring these videos also exposed a problem with our evaluation criteria. Human raters were applying several criteria inconsistently, so the same video could pass with one rater and fail with another. Finding those inconsistencies helped me strengthen the criteria guidance alongside the system fixes.

Before & after · Maze Rushsame game, same scene structure

A passing video from before the sprint, lacking pacing and coherent scene progression.

The same game after the sprint with better cinematic appeal, pacing, and story progression.

Avatar accuracy was the one gap the sprint didn’t close. Fixing it would have required bigger changes and more agents, work that would have taken more than a week. Chasing it would have pulled focus from the fixes that were raising the pass rate. So I kept the sprint on improving our existing agents, plus the one new agent we’d already planned to catch bad first frames, and slated avatar accuracy for the next release.

In 1 week, the share of videos passing our quality bar rose from 3% to 31%. Gameplay and mechanics were more accurate, storylines were more coherent, cinematics and pacing improved, avatar-style inconsistencies dropped, and far less AI slop reached the final videos.

The 4-lever framework outlasted the sprint. It became the team’s shared way to diagnose, prioritize, and solve quality problems as the system kept growing.

Next chapterChapter 3 of 3

Balancing accuracy and feel in AI-generated content →

With the pass rate at 31%, avatar accuracy was next. In the following release, one of our engineers fixed it, and avatars became far more accurate. But the videos lost the energy that made people want to play, and the cause was hidden somewhere in the instructions of more than 20 agents.

Previous chapter

← Automating video marketing assets for game developers

Case study on building the prototype for this agentic system with a research scientist.