# Grounded Reasoning Systems for Cloud Architecture

Iman Makaremi, Cat.io | AI Engineer World's Fair 2025 | 26:06

Source: https://www.youtube.com/watch?v=9mzfioh1Zag
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/grounded-reasoning-systems-for-cloud-architecture
Published: 2025-06-03
Tags: agents, evals, human-in-the-loop, multi-agent, rag

## TL;DR
- Cloud architecture copilots need to reason over requirements, architecture graphs, trade-offs, and business context rather than only automate fixed tasks.
- A multi-agent workflow uses a chief architect, specialized staff architects, retrieval agents, isolated memory, cloning, and explicit control flow to produce and refine recommendations.
- Human evaluation must be part of the system from the beginning because model confidence and LLM-based evaluation do not reliably show whether a recommendation is correct.

## Summary
Iman Makaremi describes Cat.io's AI copilot for enterprise architecture and the problems involved in making it reason over real technical environments. Architecture decisions combine textual requirements with graph-shaped infrastructure data, often involve conflicting goals, and require explanations of proposed changes. The system grounds agents with enriched component data, graph search, structured requirement extraction, and a domain-specific knowledge graph. Its orchestration uses a chief architect, specialized staff architects, requirement and architecture retrievers, structured messages, isolated agent memory, and cloned agents for parallel design proposals. The workflow generates recommendations, resolves conflicts, and writes detailed proposals with gap analysis. Makaremi is direct about the limits: semantic search loses context on graph data, larger document corpora can weaken retrieval, and confidence is not correctness. Cat.io built the Eagle Eye tool for human scoring of requirements, agent conversations, and recommendations. The talk argues that evaluation, observability, memory, and workflow control have to be designed alongside the reasoning system.

## Key ideas
### Architecture decisions require reasoning over scattered context
[00:01](https://www.youtube.com/watch?v=9mzfioh1Zag&t=1s)
Makaremi says cloud architecture has outgrown simple automation because systems, tools, constraints, and expectations keep increasing. Architects negotiate trade-offs based on requirements, available time, and resources. Much of the context is scattered and implicit, so an AI system has to understand more than infrastructure components. It must understand requirements, identify the role of each component, and recommend changes that either satisfy requirements or improve the current architecture. The underlying data has two different forms: requirements are mostly text, while architecture is graph data. Connecting those forms is necessary before an agent can handle vague questions and break them into a workable plan.

### Grounding needs both semantic retrieval and graph structure
[04:07](https://www.youtube.com/watch?v=9mzfioh1Zag&t=247s)
Cat.io enriched architecture components with semantic information so they could be found through vector search. It also added connection data and graph algorithms to retrieve related parts of an architecture. Requirements were enriched and scored against important organizational concepts to make retrieval faster. Makaremi says semantic grounding improved reasoning in some cases, especially architecture retrieval, but it could also return overly detailed results or fail to scale. Semantic search alone did not work well for graph data. The team therefore moved toward graph-based search and a knowledge graph that could connect retrieved nodes and preserve the reasons those nodes belonged together.

### Structured requirement extraction improves retrieval but can lose context
[08:30](https://www.youtube.com/watch?v=9mzfioh1Zag&t=510s)
For requirement documents, the team used templates with a defined structure. Documents were preprocessed, split, and embedded, while the templates guided the extraction of the information needed by downstream recommendation tasks. Makaremi says this made business requirements easier to structure and helped agents find relevant pieces quickly. The approach had a limit as the document set grew: larger searches sometimes lost the surrounding context. Graph analysis could help recover relationships inside a larger corpus. The lesson is that a retrieval method must preserve both the specific requirement needed for a task and the wider context that explains how it relates to other requirements.

### Multi-agent workflows need isolated memory and control flow
[10:11](https://www.youtube.com/watch?v=9mzfioh1Zag&t=611s)
The system uses specialized agents to handle conflicting goals and trade-offs. Structured message formats replaced an earlier approach based on generating and parsing XML. Agent conversations are isolated so one agent does not consume another agent's context, and the team observed more hallucination when memory increased. Agents can be cloned for parallel tasks, with each clone receiving the history available at the cloning point and then maintaining separate current history. Makaremi says more dynamic orchestration produced better creativity and planning, but agents cannot simply interact without supervision. The workflow needs explicit control flow to keep the process moving toward a usable result.

### Recommendations pass through specialist generation and conflict resolution
[14:05](https://www.youtube.com/watch?v=9mzfioh1Zag&t=845s)
The production recommendation workflow has a chief architect, 10 staff architects with domain specializations, a requirement retriever, and an architecture retriever. Staff architects first generate possible recommendations. The chief architect then checks for conflicts and redundancies and prunes the list. The remaining topics become full design proposals with a target state, gap analysis, and recommended actions. Requirement and architecture retrieval calls can run in parallel, and several staff architects can work at once. During proposal generation, staff architects are cloned according to the number of recommendations they need to produce. Each clone can use prior knowledge while keeping its new work separate from the other clones.

### Human evaluation exposes failures that confidence scores miss
[19:03](https://www.youtube.com/watch?v=9mzfioh1Zag&t=1143s)
Makaremi says the team needed human scoring, structured feedback, and revision cycles to judge whether recommendations were useful. LLM-based evaluation helped, but it did not provide enough guidance for the improvements they wanted. Cat.io built an internal tool called Eagle Eye, where reviewers can inspect the architecture, extracted requirements, agent conversations, and generated recommendations. They score qualities such as relevance, visibility, and clarity. The tool also makes individual failures visible, including a hallucinated scheduling exchange between a staff architect and the requirements retriever. Makaremi's direct conclusion is that confidence can help, but it cannot be treated as correctness. Evaluation should be designed at the same time as the AI system.

### The architecture is moving toward graphs managed by LangGraph and Flyte
[22:43](https://www.youtube.com/watch?v=9mzfioh1Zag&t=1363s)
Makaremi describes the copilot as a reasoning system with roles, workflows, memory, and structure. It must work across large architecture datasets and many documents for users ranging from developers to CTOs. After experimenting with different multi-agent frameworks, the team was moving toward LangGraph for agent workflows, with a manager coordinating those workflows. Flyte handles higher-level execution and management in the described stack, while graph storage captures as much agent memory and architectural context as possible. The design remains experimental because the best patterns depend on the data. Makaremi says graphs are becoming more important, along with decisions about agent interaction and how much autonomy each agent should receive.

## Notable quotes
- "Cloud architecture needs reasoning, not just automation." (00:24)
- "Good design involves conflicting goals, trade-offs and debates." (10:11)
- "Confidence is not correctness." (20:46)
- "Eval must be baked into system design, not added later." (20:52)
- "Building a copilot is not generating answers. It's about designing a system that can reason." (22:43)

## Tools & references mentioned
- Cat.io
- LangGraph
- Flyte
- AWS Bedrock
- Claude 3
- OpenTelemetry
- Eagle Eye

## Who should watch
- You are building an enterprise AI system that must reason over architecture data, requirements, and dependencies rather than answer questions from a short prompt.
- Your multi-agent workflow needs retrieval, memory boundaries, parallel execution, or conflict resolution, and you want an example of how those pieces fit together.
- You need an evaluation process for agent output and want to see why human review may expose problems that confidence scores and model-based graders miss.

## Related talks

- [Building Applications with AI Agents](https://aietalks.com/talks/building-applications-with-ai-agents) (Michael Albada, Microsoft, 15:50)
- [Realtime multiplayer, automation, and you!](https://aietalks.com/talks/realtime-multiplayer-automation-and-you) (Idan Gazit, GitHub, 21:41)
- [AI Copilots for Tech Architecture: The Highest-ROI Use Case You're Not Building](https://aietalks.com/talks/ai-copilots-for-tech-architecture-the-highest-roi-use-case-youre-not-building) (Boris Bogatin & Tufik Pubz, Catio, 27:29)
- [On AI and Knowledge](https://aietalks.com/talks/on-ai-and-knowledge) (Pablo Castro, Microsoft, 17:35)
- [Design Patterns for AI Trust: Juries, Libraries, and Agent Tiers](https://aietalks.com/talks/design-patterns-for-ai-trust-juries-libraries-and-agent-tiers) (Alex Bauer, Upside.tech, 17:09)