# Using AI to Build an Infinite Game Jeff Schomay | AI Engineer Summit 2023 | 11:08 Source: https://www.youtube.com/watch?v=_KFbT6eph5A Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/using-ai-to-build-an-infinite-game Published: 2024-02-01 Tags: caching, fine-tuning, multimodal, structured-outputs ## TL;DR - Jeff Schomay built a forest exploration game whose scenes, text, and images were generated by AI. - Fine-tuning a model with 50 examples let him shorten the prompt while keeping the scene JSON reliable. - Caching generated scenes hid the 10 to 30 second generation time from players. ## Summary Jeff Schomay describes a forest exploration game built with entirely AI-generated content. The player moves through scenes, encounters events, and tries to find home before losing all courage. Schomay wanted every playthrough to produce new content, so he built a pipeline that generates a scene definition with OpenAI, validates its JSON, creates a matching image with a custom Leonardo model, and sends both to the game. Fine-tuning improved the text generation enough that he could remove most of the detailed prompt. Training the image model was harder because the forest perspective, scale, and style had to stay consistent while people, animals, buildings, and empty scenes varied. Generation took 10 to 30 seconds, so he cached scenes and refilled the cache in the background. He also discusses possible extensions such as higher-resolution images, user-selected themes, weather, and time of day. ## Key ideas ### The game generates a new forest to make each playthrough different [00:01](https://www.youtube.com/watch?v=_KFbT6eph5A&t=1s) Schomay began with a simple game about wandering through a forest and finding home before courage runs out. Its original version had 16 scenes arranged in a 4x4 grid, so players would see everything after a few games. He wanted to share the pleasure of discovering what the AI created, so he changed the idea: every new scene would be generated fresh. The goal was an exploration game that could continue indefinitely, with each playthrough producing places and encounters the player had not seen before. ### A detailed prompt produced usable scene JSON, but fine-tuning made the process simpler [01:30](https://www.youtube.com/watch?v=_KFbT6eph5A&t=90s) Each scene is represented by a JSON object describing what happens when the player first finds it, what happens on a return visit, and how the encounter changes the player's stats. Schomay first used OpenAI's completion endpoint with a long prompt that specified the format and content. It worked most of the time, but he wanted more reliability. He generated 50 examples and used OpenAI's fine-tuning endpoint. Afterward, he could use a much shorter prompt without including the JSON structure. The resulting output stayed accurate, while requiring fewer tokens. ### Fine-tuning the image model required controlled variation [03:32](https://www.youtube.com/watch?v=_KFbT6eph5A&t=212s) Schomay used Leonardo to generate images and create custom image models. The training images needed the parts that should remain fixed to stay consistent, while the parts that should change had to vary. Otherwise, the model could overfit and make every image look alike, or fail to learn a shared style. His game needed the same forest setting, perspective, scale, tone, and texture across scenes. At the same time, some scenes included people, animals, or buildings, while others were empty. He trained several models with different parameters and image sets before finding a model that produced cohesive but varied results. ### Using the scene description directly made useful image prompts [04:05](https://www.youtube.com/watch?v=_KFbT6eph5A&t=245s) Schomay was surprised that the text description generated for a scene also worked well as the image prompt. The descriptions included second-person language and details that were not written like conventional image prompts, yet Leonardo produced good pictures from them. The final model generated scenes with shared visual features, including a zigzag path through the forest, consistent trees, and a common overall look. The scenes remained distinct from one another, which gave the game variety without making its world feel visually disconnected. ### The game pipeline validates text before asking for an image [06:23](https://www.youtube.com/watch?v=_KFbT6eph5A&t=383s) Schomay assembled the system in a simple asset server. It first requested a new scene from the OpenAI endpoint using his custom model. The server then validated the returned JSON to check that all required keys were present. If the data passed validation, it sent the scene description to Leonardo, which generated an image with the custom image model. The server combined the text and image and returned the completed scene to the game. A preview server let him scroll through many generated scenes and inspect whether they worked before putting them into play. ### Caching hides slow generation from the player [07:20](https://www.youtube.com/watch?v=_KFbT6eph5A&t=440s) Generating a scene and its image took 10, 20, and sometimes 30 seconds, which would interrupt play. Schomay solved this by prefilling a cache with generated scenes. When the game removed scenes from the cache, the server refilled it after the number of available scenes fell below a threshold. The player could then request a new location and receive a scene that was already prepared. This kept the generated-content idea while avoiding a long wait at every movement. ### The generated game changes both its content and its interface [08:18](https://www.youtube.com/watch?v=_KFbT6eph5A&t=498s) In the demonstration, the player always starts at the same lamppost and searches for home. Vigor controls speed, while lower courage makes the viewport smaller. Generated encounters change the player's condition. A blue glowing formation raises vigor, a campfire provides another discovery, and a dark cave lowers courage. The player can return to a previous location, but the overall game continues until home is found. Schomay says that playing again produces a different sequence because the scenes are generated anew. ### More context could make generated scenes feel tied to the player [09:57](https://www.youtube.com/watch?v=_KFbT6eph5A&t=597s) Schomay identifies several extensions to the same workflow. The images were 512 pixels wide, and an AI upscaler could improve their resolution at the cost of additional time. The prompt could also include information chosen or supplied by the player, such as a theme, the time of day, or the current weather at the player's location. The generated scenes could then reflect that context. He presents this as a process that could be reused for other games, including a physical card game mentioned in the talk description. ## Notable quotes - "I made a game with 100% AI generated content." (00:01) - "I wanted to try to do it now the first thing that I would need to do is to generate each scene and have a consistent way of doing that." (01:06) - "I was very happy to find that it worked perfectly even though I didn't mention the JSON in it at all, it came out perfect because of what was in the examples." (03:04) - "It takes 10 20 sometimes 30 seconds to do this and that wouldn't be good for the play experience." (07:20) ## Tools & references mentioned - OpenAI - OpenAI completion endpoint - OpenAI fine-tuning endpoint - Leonardo - AI upscaler ## Who should watch - You are building a game and need a large supply of inexpensive prototype content. - You want to connect text generation and image generation in one content pipeline. - You are deciding whether generation latency, model training, and caching can fit your player experience. ## Related talks - [Think You Can Build a Game with AI? Think Again!](https://aietalks.com/talks/think-you-can-build-a-game-with-ai-think-again) (Danielle An & David Hoe, Meta, 18:00) - [Books reimagined: AI to create new experiences for things you know](https://aietalks.com/talks/books-reimagined-ai-to-create-new-experiences-for-things-you-know) (Lukasz Gandecki, TheBrain.pro, 09:44) - [The Next Game Engine Won't Have a Manual](https://aietalks.com/talks/the-next-game-engine-wont-have-a-manual) (Arturo Nunez, Nereu, 19:33) - [Building a Chess Coach](https://aietalks.com/talks/building-a-chess-coach) (Anant Dole & Asbjørn Steinskog, Take Take Take, 18:22) - [Fast Models Need Slow Developers](https://aietalks.com/talks/fast-models-need-slow-developers) (Sarah Chieng, Cerebras, 18:02)