An agent can turn Minecraft chat requests into tool calls that move, locate, dig, and interact with objects in the game.
2
Amazon Bedrock Agents with Claude 3 Haiku replaced an earlier LangChain and SageMaker setup as Rocky's agent workflow became more complex.
3
A build experiment uses Claude to convert a structure description into JSON coordinates that Mineflayer can use to place Minecraft blocks.
Summary
Derek demonstrates Rocky, an open-source Minecraft bot controlled through the game's chat. Rocky uses Minecraft actions as tools, so a player can ask it to come to a location, find a pig, hit the pig, or dig a hole with dimensions such as 2 by 2. Derek explains how the project moved from a container running Minecraft and Mineflayer, with LangChain on Lambda and a model hosted in SageMaker, to Amazon Bedrock Agents. The current setup uses Claude 3 Haiku and Bedrock's return-control feature. The talk also shows an experimental build action. Claude receives a description such as "double decker couch" and returns JSON describing blocks and coordinates. Mineflayer then builds that structure in the game. Derek is open about the prompt rules needed to keep the model from producing unusable structures, and he shares the project as open source.
Minecraft chat gives the agent its requests and the game provides its tools
Derek begins with an agent workflow in which an agent consumes an input, calls one or more tools, and may repeat that process through an LLM before returning a response. In Minecraft, the input comes from the game's chat function. The tools are actions available in the world, such as building, digging, farming, moving, and locating players. Rocky is the bot that receives the chat request and tries to carry it out. Derek points out that both the request and the result can be unpredictable, which makes a live LLM demo risky.
Rocky can carry out simple actions and find entities in the game
The recorded demo shows Rocky answering a weather question, jumping, finding Derek after he hides behind a building, and locating a pig. When Rocky finds the pig, it reports the location and can then move to it. Derek also asks Rocky to hit the pig. The bot can dig a hole with parameters supplied in the request, such as a 2 by 2 hole, and it can later find its way out. Derek says that Rocky digging its way out was behavior the team did not expect, but it worked in the game.
The architecture changed as the number of actions grew
The project started with Minecraft and Mineflayer running together in a container because Mineflayer needs to keep state. Derek says Mineflayer cannot run on Lambda for that reason. The team first used LangChain on Lambda to build the agent and Amazon SageMaker to host Cohere models. As Rocky gained more actions and tools, that setup became difficult to manage. The version shown in the talk keeps the Minecraft server and Mineflayer client running locally for the demo and calls an agent built with Amazon Bedrock.
Bedrock provides the managed model and agent workflow
Derek describes Amazon Bedrock as a common API over several model providers, including Anthropic, Cohere, and Meta's Llama models. Bedrock also provides agents, knowledge bases for retrieval-augmented generation, guardrails, evaluations, prompt creation, and tracing. Rocky uses Claude 3 Haiku because Derek wanted a fast model. The agent can be traced in the console or through logs. Return control is especially important for this project because Bedrock hands an action back to the application, which then performs that action in Minecraft.
Rocky's actions use return control and can accept parameters
In the Bedrock console, Derek shows the Minecraft agent, its Claude 3 Haiku model, and its system prompt. The prompt describes Rocky as a playful, friendly, creative Minecraft agent. The action definitions include jump, dig, and other operations, and they use return control so the Minecraft client receives the requested action. Dig has parameters such as depth and width. A request for a small hole is interpreted as a one by one hole, while a request for a 2 by 2 hole supplies larger dimensions.
The experimental build action translates descriptions into Minecraft geometry
Derek adds a Minecraft experiment action called build. Building is harder than the simpler actions because Minecraft is a three-dimensional space. In the live demo, he asks Rocky to build a double-decker couch. The build prompt tells Claude to act as an expert Minecraft builder and to return valid JSON. That JSON describes the blocks and their X, Y, and Z coordinates in the format Mineflayer uses. The model receives rules about which blocks it may use and how blocks should be placed next to one another.
Strict output rules keep the generated structures usable
Derek says the build prompt needs strict rules because, without them, the model produces nonsense. The prompt requires valid JSON, limits the blocks that can be used, and explains the coordinate format. Claude interprets the phrase "double decker couch" in Minecraft terms, and Mineflayer places the resulting blocks. The finished structure is Rocky's interpretation of that object in the game's 3D space. Derek presents this as an experimental feature alongside the simpler movement and digging actions.
"The tools in Minecraft are many and that's why we built this in Minecraft."01:33
Who should watch
You are building an agent that needs to call tools and hand control back to an application that owns the state.
You want a concrete example of connecting an LLM to a changing game world rather than returning text alone.
You are comparing LangChain and SageMaker with a managed Amazon Bedrock Agents setup, especially for AWS serverless applications.
