AI agents stay useful when they learn from production data, user feedback, and changing enterprise knowledge.
2
A data flywheel can fine-tune smaller models to approach the accuracy of a much larger model while reducing latency and inference cost.
3
NVIDIA's NeMo microservices provide components for data curation, customization, evaluation, guardrails, and retrieval across an agent system.
Summary
Sylendran Arunagiri argues that improving an AI agent does not require continually adopting larger language models. Agents need a feedback loop that collects production data, user feedback, and business knowledge, then uses that information to curate ground truth, evaluate models, fine-tune smaller models, and promote the models that meet the application's accuracy target. He explains this approach through NVIDIA's NeMo microservices and an internal employee support agent called NV Info. In the routing example, an untuned 70B model reached 96% accuracy, while an 8B model reached less than 14%. After the team collected and analyzed feedback, fine-tuned smaller models, and evaluated them against a curated dataset, an 8B model matched the 70B model's accuracy. A 1B model reached 94% accuracy with much lower cost and latency. The talk ends with a four-part framework: monitor feedback, analyze and attribute failures, plan experiments, and execute a regular production evaluation and retraining cycle.
An agent becomes useful over time when it learns from feedback
Sylendran Arunagiri defines agents as systems that perceive, reason, and act through tools and external systems. He adds another part to the cycle: capturing user feedback and learning from user preferences and data. That feedback lets an agent become more accurate and useful for its task. This matters because enterprise data changes quickly, business intelligence keeps arriving, and user needs change. An agent that only runs a fixed model and fixed retrieval setup can become less helpful as its environment changes.
A data flywheel connects production signals to model improvement
The data flywheel starts with enterprise data and forms a continuous cycle of data processing and curation, model customization, evaluation, guardrails, and retrieval. Once an agent runs in production, inference data, business intelligence, and user feedback help create ground truth. The team can then experiment with existing and newer models, evaluate them, and find smaller models that meet the accuracy expected from larger models. Those smaller models can offer lower latency, faster inference, and lower total cost of ownership.
NeMo microservices cover the main stages of the loop
NVIDIA's NeMo microservices include NeMo Curator for high-quality training data, including multimodal data; NeMo Customizer for methods such as LoRA, P-Tuning, and full SFT; NeMo Evaluator for academic, institutional, and LLM-as-a-judge evaluations; NeMo Guardrails for privacy, security, and safety; and NeMo Retriever for retrieval-augmented generation pipelines. Arunagiri says these components are exposed through APIs and can run on-premises, in the cloud, in a data center, or at the edge.
The sample architecture promotes a model only after evaluation
In the sample architecture, an end user interacts with a guarded agent backed by a model served through NVIDIA NIM for optimized inference. Data is curated and stored in NeMo Data Store. NeMo Customizer and NeMo Evaluator support repeated retraining and evaluation. When a model reaches the target accuracy, an IT administrator or AI engineer can promote it to power the agent. Arunagiri describes the microservices as Lego pieces that can be assembled into a complete data flywheel.
NV Info uses routing, expert agents, and retrieval
NVIDIA applied the approach to NV Info, an internal employee support agent. The system answers questions about HR benefits, financial earnings, IT help, product documentation, and other enterprise topics. A router agent interprets the query and sends it to a domain-specific expert agent. Each expert uses a retrieval pipeline to find relevant information. Production inference logs, employee feedback, and subject matter expert review continuously provide material for the flywheel.
Feedback analysis turned a routing problem into training data
The team compared models for routing queries to expert agents. An untuned 70B model reached 96% routing accuracy, while an 8B model scored below 14%. They then collected 1,224 employee feedback data points, with 729 satisfactory and 495 unsatisfactory responses. NeMo Evaluator and an LLM judge helped investigate the unsatisfactory cases. The team identified incorrect routing in 140 cases, then found 32 cases that were truly caused by incorrect routing through subject matter expert review. They built an 865-point ground-truth dataset and split it 60/40 for fine-tuning and evaluation.
Fine-tuning changed the cost and latency trade-off
The 70B model achieved 96% accuracy but took 26 seconds to generate its first token. The untuned 8B model had 14% accuracy, with latency about 70% lower. After fine-tuning, the 8B model matched the 70B model's accuracy. A 1B model reached 94% accuracy, only 2% below the 70B result. Arunagiri says the 1B option offered 98% lower inference cost, a 70x reduction in model size, and 70x lower latency. The choice depends on how much accuracy, cost, and computing resource the use case requires.
A practical flywheel needs feedback, diagnosis, planning, and execution
Arunagiri's framework starts with monitoring user feedback through implicit and explicit signals while respecting privacy. Teams then analyze and attribute failures or model drift, classify errors, and create a ground-truth dataset. During planning, they identify candidate models, generate synthetic datasets, run experiments, fine-tune models, and assess resource and cost needs. Execution includes triggering the flywheel and setting a regular cadence for tracking accuracy, latency, production logs, and end-to-end GenAIOps performance.
"What completes the cycle is if AI agents are able to capture and learn from user feedback along the way."01:27
Who should watch
You are running an agent whose enterprise data, user preferences, or production behavior changes after launch.
Your team is considering a large model for routing or another narrow agent task and needs a way to test smaller fine-tuned alternatives.
You need a concrete operating process for collecting feedback, diagnosing failures, evaluating models, and managing recurring retraining.
