# Building an Agentic Platform

Ben Kus, Box | AI Engineer World's Fair 2025 | 19:06

Source: https://www.youtube.com/watch?v=12v5S1n1eOY
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/building-an-agentic-platform
Published: 2025-08-24
Tags: agents, evals, multimodal, workflows

## TL;DR
- Box moved from single-pass LLM extraction to agentic workflows after complex documents exposed limits in OCR, context handling, and model confidence.
- The agentic system prepares and groups fields, runs multiple document queries, checks results with tools and other models, and sends uncertain answers back for another pass.
- Ben Kus argues that teams should build the agentic architecture early when AI models could plausibly help solve the problem, because the workflow can evolve without replacing the whole system.

## Summary
Ben Kus describes how Box built its AI platform through metadata extraction. The first generative AI approach used OCR, prompts, and single calls to extract fields from documents. It worked well on simple cases, then struggled with long documents, many fields, difficult OCR, multiple languages, and complex questions such as risk assessment. Box rebuilt the process as an agentic workflow. The system groups related fields, breaks work into multiple queries, uses OCR and page images to check results, compares outputs from different models, and uses a judge to provide feedback for another attempt. Kus says this architecture made it easier to fix new problems by adding or changing workflow steps. The same foundation later supported deep research across Box content. He separates the agentic workflow layer from the distributed systems work needed to process content at scale. Box currently prefers prompts, cached prompts, and agentic workflows over fine-tuning because it supports Gemini, Llama, OpenAI, and Anthropic models and would otherwise need to maintain fine-tunes across model versions.

## Key ideas
### Enterprise content gave Box a large extraction problem
[00:01](https://www.youtube.com/watch?v=12v5S1n1eOY&t=1s)
Box focuses on large enterprises, with more than 115,000 enterprise customers and two-thirds of the Fortune 500. Kus says the platform stores more than an exabyte of data and hundreds of billions of files. Most enterprise data is unstructured, including contracts and project proposals, while only a smaller share sits in databases. Extracting fields from that content would support search, filtering, queries, and workflows. The older industry approach required specialized models, focused content types, large training corpora, and custom machine-learning work, which made extraction brittle and limited its use.

### Single-pass LLM extraction worked until customers supplied real complexity
[04:54](https://www.youtube.com/watch?v=12v5S1n1eOY&t=294s)
Box first combined standard preprocessing and OCR with prompts that asked a general model to extract fields. This was flexible and often outperformed older specialized systems. The limits appeared when customers asked about 300-page leases with 300 fields, complex digital assets, or risk assessments. OCR errors, handwriting, crossed-out text, file formats, and languages could all damage the input. Models also lost track when many fields had separate instructions. An LLM judge could warn that an answer might be wrong, but customers wanted the extraction to work rather than receive a confidence warning.

### Agentic workflows let extraction split, check, and retry work
[09:30](https://www.youtube.com/watch?v=12v5S1n1eOY&t=570s)
Box responded with an agentic design built around instructions, objectives, model access, secure tools, memory, and a directed graph. The inputs and outputs stayed the same, with documents entering and fields or answers coming out. Inside the workflow, Box prepares and groups fields, because related information such as contract parties and their addresses needs to be handled together. It then runs multiple document queries and checks results with tools. Some checks use OCR or page images. Multiple models can vote, and a judge can give feedback that sends the process back for another attempt.

### The architecture made new fixes local changes instead of rewrites
[12:28](https://www.youtube.com/watch?v=12v5S1n1eOY&t=748s)
Kus says the agentic approach changed how Box handled new problems. A team could add another double-check or adjust a prompt for one part of the process instead of redesigning the entire extraction system. The same pattern applied to document question answering. If a judge found that an answer was weak, the system could ask the model to reflect and try again before showing it to the user. Box also used the foundation for deep research over customer content, with steps for searching, checking relevance, making an outline, and carrying out the research process.

### Agentic workflows and large-scale infrastructure should stay separate
[14:05](https://www.youtube.com/watch?v=12v5S1n1eOY&t=845s)
Kus describes the agentic abstraction as a directed, intelligent workflow in which models power individual steps. He separates that layer from the distributed systems work needed to process very large volumes of content. The two problems have different concerns. One team can focus on how an agent reasons through a task, while another focuses on scaling a generic process across huge document collections. He also says the workflow is easy to evolve. When deep research produced sloppy output, Box added a final instruction to produce the desired format rather than rebuilding the system.

### Box wants teams to build the agentic layer early
[16:05](https://www.youtube.com/watch?v=12v5S1n1eOY&t=965s)
Kus says the lesson became clear only after Box had spent time developing the platform. When a set of AI models could plausibly help solve a problem, he would now build the agentic architecture early. This gives the team a place to add tools, checks, prompts, and workflow steps as requirements change. He also wants engineers to think about the platform from the perspective of customers building their own systems, including the tools exposed through MCP servers and agent-to-agent communication.

### Box currently chooses prompts and multiple models over fine-tuning
[18:27](https://www.youtube.com/watch?v=12v5S1n1eOY&t=1107s)
In the questions, Kus explains that Box is currently against fine-tuning for these use cases. Once a model is fine-tuned, each later model version may require another fine-tune. Box supports Gemini, Llama, OpenAI, and Anthropic models, so keeping behavior consistent across vendors and versions would be difficult. The company instead uses prompts, cached prompts, and agentic workflows. Box evaluates agents with standard evaluation sets, harder challenge sets, an LLM judge, and customer feedback.

## Notable quotes
- "We ended up with a set of features, things like QA across documents, things like being able to extract data, things like being able to do AI powered workflows." (01:45)
- "The fragility of the architecture was one that we weren't really going to be able to solve on our own." (09:03)
- "The answer was not rethink everything or let's then try like a whole new set of things." (12:29)
- "If it's plausible that a set of AI models could help you solve that problem, then you should build this AI agentic architecture early." (16:05)
- "We've got to the point where we use prompts or cache prompts or agenticness as opposed to fine-tuning." (18:40)

## Tools & references mentioned
- Box
- Gemini
- Llama
- OpenAI
- Anthropic
- LangGraph
- MCP
- IBM
- Subspace
- BigFix

## Who should watch
- You are building document extraction and single-pass LLM calls are breaking on long, multilingual, or field-heavy documents.
- Your team needs an architecture that can add checks, retries, and tool calls without replacing the whole workflow.
- You are deciding between fine-tuning and a model-independent agentic approach for a product that must support several model vendors.

## Related talks

- [Building AI Agents that Actually Automate Knowledge Work](https://aietalks.com/talks/building-ai-agents-that-actually-automate-knowledge-work) (Jerry Liu, LlamaIndex, 17:57)
- [The Age of the Agent](https://aietalks.com/talks/the-age-of-the-agent) (Flo Crivello, Lindy, 14:49)
- [Agentic Enterprise: What Your CEO Must Know About AI](https://aietalks.com/talks/agentic-enterprise-what-your-ceo-must-know-about-ai) (Hubert Misztela, Novartis, 28:04)
- [Disrupting the $15 Trillion Construction Industry with Autonomous Agents](https://aietalks.com/talks/disrupting-the-15-trillion-construction-industry-with-autonomous-agents) (Dr. Sarah Buchner, Trunk Tools, 05:32)
- [Platforms for Humans and Machines: Engineering for the Age of Agents](https://aietalks.com/talks/platforms-for-humans-and-machines-engineering-for-the-age-of-agents) (Juan Herreros Elorza, Banking Circle, 21:15)