Jeff Schomay built a forest exploration game whose scenes, text, and images were generated by AI.
2
Fine-tuning a model with 50 examples let him shorten the prompt while keeping the scene JSON reliable.
3
Caching generated scenes hid the 10 to 30 second generation time from players.
Summary
Jeff Schomay describes a forest exploration game built with entirely AI-generated content. The player moves through scenes, encounters events, and tries to find home before losing all courage. Schomay wanted every playthrough to produce new content, so he built a pipeline that generates a scene definition with OpenAI, validates its JSON, creates a matching image with a custom Leonardo model, and sends both to the game. Fine-tuning improved the text generation enough that he could remove most of the detailed prompt. Training the image model was harder because the forest perspective, scale, and style had to stay consistent while people, animals, buildings, and empty scenes varied. Generation took 10 to 30 seconds, so he cached scenes and refilled the cache in the background. He also discusses possible extensions such as higher-resolution images, user-selected themes, weather, and time of day.
The game generates a new forest to make each playthrough different
Schomay began with a simple game about wandering through a forest and finding home before courage runs out. Its original version had 16 scenes arranged in a 4x4 grid, so players would see everything after a few games. He wanted to share the pleasure of discovering what the AI created, so he changed the idea: every new scene would be generated fresh. The goal was an exploration game that could continue indefinitely, with each playthrough producing places and encounters the player had not seen before.
A detailed prompt produced usable scene JSON, but fine-tuning made the process simpler
Each scene is represented by a JSON object describing what happens when the player first finds it, what happens on a return visit, and how the encounter changes the player's stats. Schomay first used OpenAI's completion endpoint with a long prompt that specified the format and content. It worked most of the time, but he wanted more reliability. He generated 50 examples and used OpenAI's fine-tuning endpoint. Afterward, he could use a much shorter prompt without including the JSON structure. The resulting output stayed accurate, while requiring fewer tokens.
Fine-tuning the image model required controlled variation
Schomay used Leonardo to generate images and create custom image models. The training images needed the parts that should remain fixed to stay consistent, while the parts that should change had to vary. Otherwise, the model could overfit and make every image look alike, or fail to learn a shared style. His game needed the same forest setting, perspective, scale, tone, and texture across scenes. At the same time, some scenes included people, animals, or buildings, while others were empty. He trained several models with different parameters and image sets before finding a model that produced cohesive but varied results.
Using the scene description directly made useful image prompts
Schomay was surprised that the text description generated for a scene also worked well as the image prompt. The descriptions included second-person language and details that were not written like conventional image prompts, yet Leonardo produced good pictures from them. The final model generated scenes with shared visual features, including a zigzag path through the forest, consistent trees, and a common overall look. The scenes remained distinct from one another, which gave the game variety without making its world feel visually disconnected.
The game pipeline validates text before asking for an image
Schomay assembled the system in a simple asset server. It first requested a new scene from the OpenAI endpoint using his custom model. The server then validated the returned JSON to check that all required keys were present. If the data passed validation, it sent the scene description to Leonardo, which generated an image with the custom image model. The server combined the text and image and returned the completed scene to the game. A preview server let him scroll through many generated scenes and inspect whether they worked before putting them into play.
Generating a scene and its image took 10, 20, and sometimes 30 seconds, which would interrupt play. Schomay solved this by prefilling a cache with generated scenes. When the game removed scenes from the cache, the server refilled it after the number of available scenes fell below a threshold. The player could then request a new location and receive a scene that was already prepared. This kept the generated-content idea while avoiding a long wait at every movement.
The generated game changes both its content and its interface
In the demonstration, the player always starts at the same lamppost and searches for home. Vigor controls speed, while lower courage makes the viewport smaller. Generated encounters change the player's condition. A blue glowing formation raises vigor, a campfire provides another discovery, and a dark cave lowers courage. The player can return to a previous location, but the overall game continues until home is found. Schomay says that playing again produces a different sequence because the scenes are generated anew.
More context could make generated scenes feel tied to the player
Schomay identifies several extensions to the same workflow. The images were 512 pixels wide, and an AI upscaler could improve their resolution at the cost of additional time. The prompt could also include information chosen or supplied by the player, such as a theme, the time of day, or the current weather at the player's location. The generated scenes could then reflect that context. He presents this as a process that could be reused for other games, including a physical card game mentioned in the talk description.
"I was very happy to find that it worked perfectly even though I didn't mention the JSON in it at all, it came out perfect because of what was in the examples."03:04
Who should watch
You are building a game and need a large supply of inexpensive prototype content.
You want to connect text generation and image generation in one content pipeline.
You are deciding whether generation latency, model training, and caching can fit your player experience.
