Automating video marketing assets for game developers

My research scientist partner owned the technical architecture. I owned everything the system needed to do. That meant the system prompts handed from agent to agent to take a game, understand it, and turn it into a video, plus the criteria for judging whether the video was any good. My partner built the scaffolding, and I designed the behavior that ran on it.

While my partner wired external models together on internal infrastructure, I was running scrappy tests right on the websites for the models he was exploring at the time — proving out how you could chain an LLM's output into image generation, then into video generation, and then into stitching multiple clips into one coherent video.

I used Llama to write and iterate what became the system prompts for the storyline and scene descriptions. I fed its outputs into Flux, with hand-picked game images, to generate the first and last frame of each scene. Then I fed those frames and prompts into Midjourney to generate each clip, and worked out by hand what the system needed to score a soundtrack and stitch the scenes into a final video.

Each handoff showed me what the next agent and its underlying model needed. That's context engineering in practice. Deciding what information each step gets, and in what shape.

I'm not a video expert, so I borrowed one. I recruited an in-house video producer who wrote storyboards for Horizon's professionally produced game trailers. I asked him for a few storyboard templates. A template might call for 16 seconds and four scenes with fast, energetic music. First a wide third-person action shot, then a close shot of the core mechanics, then a celebration, then a still of the game's title slate.

I turned each template into machine-readable instructions for our scene-planning agent, which generated a script and scene plan that every other agent built from. The scene plan was the context the rest of the system depended on.

16 seconds — 4 scenes

Music — Fast and energetic

Scene 1 — 3rd person wide shot action cam
  • full gameplay action
  • 4 seconds
  • cut — instant

Scene 2 — close shot / insert shot
  • core gameplay / mechanics
  • 4 seconds
  • cut — instant

Scene 3 — close shot celebration
  • celebration
  • 4 seconds
  • cut — instant

Scene 4 — still game slate image
  • game title image
  • 4 seconds

I also set the creative and technical requirements for the game assets the system took in, including images, gameplay captures, and metadata. That work became the backbone of our input-asset pipeline, the follow-on system for gathering game assets our agentic video-generation system could use.

One core piece of the input-asset pipeline was a game description generator that took a game's listing description and metadata and produced a description tailored specifically for creating a video short about the game.

When something failed in the generation process, I analyzed the JSON data in the session logs to see what each agent handed off to the next agent in the workflow. This helped me understand the instructions different agents wrote based on my instructions, so I could debug where in the system something failed and why. At the time, it was a lot of system sleuthing by hand, a task I'd now use Claude Code to do. But it gave me a deep technical foundation of the system as a whole, allowing me to help others understand it and pinpoint issues as it became more complex over time.