General-purpose LLMs lose track of physical structure when they must reason over hundreds of thousands of industrial sensors and equipment names.
2
A planner LLM paired with deterministic tree traversal, pattern matching, and set operations keeps query cost roughly constant as the system grows.
3
Production systems should leave ambiguous language and judgment to the LLM while moving retrieval, counting, deduplication, and other structured work into code.
Summary
Raahul Singh and Vanč Levstik describe deploying LLMs against AI-factory and data-center systems with hundreds of thousands of sensors, equipment nodes, and inconsistent naming conventions. At small scale, putting equipment names into the context works. At larger scale, vector search confuses near-identical names, long outputs trigger frequency penalties, and sharding causes hallucinated or missing equipment. Their design uses the physical hierarchy of the facility. An LLM interprets the user's request and produces a structured search plan, while deterministic software resolves locations, applies filters, and performs set operations. The LLM sees compact path summaries or naming patterns instead of every node. In their tests, the old approach fell from 80% correctness at 64 GPUs to about 30% at 460,000 GPUs, while the new approach reached 100% on their tests and had zero failures across 66 cases on six production systems. The speakers connect this design to a move from prompt-heavy prototypes toward deterministic software in production.
Equipment names become ambiguous long before the physical system does
Phaidra's agents answer questions about individual equipment, groups of equipment, and conditions across data halls. Customers use inconsistent naming schemes, from names that expose a rack or data hall to opaque strings such as "CH3 something something six." A small language model can often infer the intended equipment when the whole system is small. At one-gigawatt factory scale, however, more than 400,000 GPUs bring power meters, chillers, and other equipment with them. The context window fills quickly. The speakers distinguish a demo from a product: a demo needs to work in one case, while a product must work across scenarios without silently failing.
Vector search and long generated lists fail on repeated industrial names
Embedding-based retrieval struggles when names differ by only one character, such as Chiller 6 and Chiller 7. Their vectors can be too similar to provide accurate recall. The model also has frequency penalties that affect repeated tokens. When asked to list many GPUs in a location, it may interpret the repeated naming pattern as a degenerate loop and stop producing output. The underlying data can be correct while the model still shuts off its response. The speakers therefore reject both naive retrieval-augmented generation and direct prompting as sufficient approaches for production-scale equipment lookup.
Parallel sharding creates missing equipment and invented equipment
A natural response to scale is to divide the equipment into shards and make parallel LLM calls. In their tests, this produced poor recall and hallucinations. Models invented phantom equipment that did not exist and silently dropped equipment that did. That behavior is unacceptable when users need an exact account of a mission-critical system. Missing a particular problem can also allow it to cascade into larger failures. The system therefore cannot grow in proportion to every component or node. The speakers argue that the design should grow with the depth of the physical hierarchy, which increases slowly even when the number of leaf devices increases sharply.
A compact hierarchy lets the model reason over structure instead of every node
An AI factory can be represented as a tree: data centers contain data halls, which contain aisles, rows, racks, and GPUs. Chiller plants have their own hierarchy of rooms, chillers, pumps, and cooling towers. The width of these trees can grow very quickly, but the path from the root to a device has limited depth. Their linearizer gives the LLM a summarized representation of the graph, describing the paths through the system. A factory with more than a million nodes can still be represented through a finite list of paths. The speakers say that a 64-GPU system and a 460,000-GPU system produce roughly the same summary size when represented this way.
The LLM should plan a search while deterministic software performs it
The LLM is used to interpret a request such as finding GPUs running hot in data hall 11. It returns a structured plan containing the equipment type, the subtree to search, and the filter to apply. A deterministic resolver then retrieves the relevant pre-indexed subsets and applies the filter. For example, it can intersect the set of GPUs in data hall 11 with the set of GPUs that are running hot. Set operations provide exact recall without requiring the model to inspect every equipment name. The system can also use patterns for vague requests. The LLM identifies a naming or data pattern, and backend code executes that pattern over the full system.
The two- or three-step resolver keeps query cost flat
The end-to-end flow starts with a user query, passes through a planner LLM, and then uses a deterministic resolver to create the result set. The resolver indexes data, performs set operations, and maps the result to the user's request. This takes two or three steps rather than an agentic loop that repeatedly searches and revises its work. The speakers say this keeps total cost relatively flat. The LLM receives compact summaries and produces a search plan, while the backend handles the large equipment inventory. That division avoids making token use grow linearly with the equipment count.
Production tests showed a large accuracy and token gap
Vanč Levstik describes head-to-head tests using the same LLM model and data for the old and new systems, with three runs per case. The old approach reached 80% correctness at 64 GPUs and fell to about 30% at 460,000 GPUs. The new approach maintained 100% accuracy across those tests. It also produced zero failures in 66 cases from six real production systems. At one-gigawatt scale, a single validation pass with the old approach used 116 million tokens. The new approach used 390,000 tokens. Query cost stayed at 9,000 tokens for systems containing either 64 GPUs or 460,000 GPUs.
Production AI systems should move structured work from prompts into code
The speakers use Andrej Karpathy's framing of software 1.0 as deterministic code and software 3.0 as behavior prompted from an LLM. LLMs are useful for parsing ambiguous requests, deciding where to look, handling unfamiliar wording, and writing the final answer. Structured work belongs in code. They specifically name retrieval, exact set logic, counting, deduplication across near-identical names, and anything that must be reproducible. Their development process began with a mostly prompt-based prototype because that quickly exposed what needed to be built. As scale requirements became clear, they moved known engineering problems into deterministic components while leaving judgment to the model.
"The simple heuristic that usually works, if you can write down the structure or the rules, it's a 1.0 job."14:37
Who should watch
You are building an LLM interface over a large industrial, data-center, or infrastructure graph and are seeing wrong or missing entity matches.
Your prototype works when the inventory is small, but context size, latency, repeated names, or multi-step agent loops are becoming production problems.
You need a practical boundary between model judgment and deterministic backend logic for systems where retrieval and counts must be exact.
