Enterprise conversation analysis needs complete coverage because manual sampling misses problems that are developing across large volumes of interactions.
2
LLM applications need a measurement and feedback system before enterprise customers can trust their outputs.
3
Echo AI combines human review, automated feedback, model monitoring, and prompt and model changes to maintain accuracy for different customers.
Summary
Trey Doig describes Echo AI's platform for analysing customer conversations across support, sales, and success teams. The company gathers conversations from different systems, cleans them, and runs configurable LLM pipelines to find intent, sentiment, operational problems, and unexpected themes. He argues that manual review and scripted analysis either cover too little or only find problems the business already knows to look for. Arjun Bansal then explains Log10's approach to measuring and improving LLM accuracy. Human review is expensive, while LLM-as-a-judge systems can show positional, verbosity, and self-preference biases. Log10 builds automated feedback models from labelled examples, fine-tuning, and synthetic data. In one experiment, 50 labelled examples reached accuracy close to that of 1,000 examples. Echo AI uses the system to grade summaries, accept human overrides, detect model drift, and improve fine-tuning datasets. The talk is candid that enterprise deployment depends on ongoing evaluation and customer-specific criteria, not just putting an LLM behind an application.
Large customer teams lose direct knowledge of conversations as interaction volume grows
Trey Doig says customer support, sales, and success conversations contain information about what customers need and where the business is failing. At a manageable scale, staff can review conversations and pass those insights to the company. As the customer base grows, the volume becomes too large and the company loses touch with what customers are saying every day. Routine issues may be visible, but problems underneath the familiar categories remain unseen. Echo AI applies this idea to high-volume customer interactions, where every conversation may contain information that a business cannot extract through ordinary human effort.
Manual sampling and scripted analysis find only part of what is happening
Enterprise teams often review a small sample of conversations, perhaps checking compliance, agent performance, or recurring subjects. Doig says this leaves people dissatisfied because they know the process is not very accurate. Engineers may then write scripts and combine data from several systems for retrospective analysis. The next step is software that searches every conversation for specific, predefined issues. That gives the company more coverage, but it still looks for things people already know to search for. Doig compares the result to discovering a fire after it has formed, without knowing where the smoke is.
Generative AI can examine every conversation and surface issues the business did not define in advance
Doig says generative AI makes 100% coverage possible. Instead of sampling conversations or searching only for known categories, the system can inspect everything and surface unexpected patterns. From one customer message, Echo AI can extract the customer's intent, the business issue behind it, and the sentiment of both the customer and the representative. In an example involving Wine Enthusiast, the platform surfaced a manufacturing defect in newly shipped wine refrigerators. The company could address the problem in real time instead of allowing it to continue for weeks and become larger.
Enterprise deployment starts with data preparation and customer-defined evaluation criteria
Echo AI first gathers conversations from contact systems, ticket systems, and other channels. It normalizes and cleans the data before compressing it into prompts. Doig says the platform then runs dozens of configurable pipelines, with customers defining what they care about and helping write the prompts. Customers eventually manage those prompts themselves. This setup matters because every customer has different questions and quality requirements. The company also self-hosts some models because its workloads involve many prompts and high throughput across different customer domains.
Trust requires measuring accuracy and working toward a target with the customer
Doig says enterprise customers are especially concerned about whether they can trust generated insights more than the work of analysts or business leaders. Echo AI tries to establish that trust within seven days by showing a new customer its initial insights. It then works toward what Doig calls 95% accuracy through sampling and investigation. Log10 helps the company's engineering and implementation teams manage this process when a customer says an output is not right. The goal is an ongoing process where customer feedback can improve the system rather than a one-time accuracy check.
Automated feedback can reduce the cost of evaluating LLM outputs
Arjun Bansal presents Log10 as an infrastructure layer for improving LLM application accuracy. Human review is treated as the deployment standard, but Bansal says it is time-consuming and expensive. LLMs used as judges introduce their own problems, including a preference for their own outputs, positional bias, and verbosity bias. Log10's Auto Feedback system is intended to produce a closer correlation between predicted feedback and actual human feedback. That signal can support monitoring, help teams send limited human review capacity to the right cases, and curate datasets for prompt improvement and fine-tuning.
Small labelled datasets can train useful evaluation models
Log10 tested several ways to build Auto Feedback models. The approaches used ground-truth datasets containing inputs, outputs, a grading rubric, and human feedback. The team compared few-shot learning, fine-tuning, and fine-tuning with bootstrap-generated synthetic data. For a summarization task, annotator-specific models produced a 45% improvement in evaluation accuracy over aggregate models. Moving from GPT-3.5 to GPT-4, switching from few-shot learning to fine-tuning, and using synthetic data also improved results. Bansal says the bootstrap approach reached accuracy close to 1,000 labelled examples using 50 ground-truth examples.
Evaluation becomes part of application maintenance after deployment
In Echo AI's demo, summaries and other conversation insights are graded against criteria defined by the team. An engineer can inspect why a point was deducted and override the score with human feedback. Doig says this gives solution engineers and developers a low-effort way to collect high-fidelity labels for fine-tuning datasets. The same process exposed a failed summary and let the team inspect the underlying instructions. Echo AI has used it to track quality regressions and model drift in a data-driven way, instead of relying only on humans sampling outputs.
"We've been able to track regressions via this process, we've been able to see model drift in a meaningful way and in a data-driven way that we previously were unable to do."18:58
Who should watch
You are building a customer-facing LLM application and need a way to evaluate outputs before customers rely on them.
Your team has too many support or sales conversations for manual review to provide useful coverage.
You need to connect human corrections to prompt changes, fine-tuning data, model monitoring, or customer-specific quality criteria.
