Method Financial replaced a slow, manual process for collecting financial liability data with an AI workflow built around GPT-4.
2
GPT-4 worked well enough to keep in production, but its cost, latency, prompt maintenance, and hallucinations prevented it from scaling.
3
Fine-tuning a smaller model produced the quality, latency, and cost profile Method needed for high-volume production use.
Summary
Mustafa Ali explains how Method Financial automated the collection of detailed liability data, such as auto-loan payoff amounts and mortgage escrow balances. The old process relied on offshore contractors calling banks, authenticating users, checking responses, and entering the results. GPT-4 handled the unstructured-data problem well, but its $70,000 first-month bill, slow responses, changing prompts, and hard-to-catch errors created a new scaling problem. Method needed a system capable of 16 million requests per day, 100,000 concurrent load, and sub-200-millisecond latency. Kyle Corbitt describes benchmarking model error rates, latency, and cost before fine-tuning. OpenPipe used Method's production data and outputs from o3-mini to fine-tune an 8-billion-parameter Llama 3.1 model. The smaller model beat GPT-4 on the task, met Method's error threshold, reduced latency, and lowered cost. The speakers argue that fine-tuning should follow production benchmarking, and that two engineers can manage the work when the use case and data are already clear.
Method turned a manual bank-calling process into an AI data workflow
Method Financial collects liability data from credit bureaus, card networks such as Visa and Mastercard, financial institutions, and other sources. Its customers wanted details such as an auto loan's payoff amount or a mortgage's escrow balance. There was no central API for these fields, and direct bank integrations could take years. Customers were using offshore contractors who called banks, authenticated with them, gathered information, proof-checked it, and entered it into financial platforms. Mustafa describes this as slow, expensive, synchronous, and prone to human error.
GPT-4 solved the extraction problem before creating a scaling problem
Method built an agentic workflow with GPT-4 to parse unstructured financial information. It worked well for extraction, classification, and summarization, and the team expanded each call to cover more use cases because API costs were high. The first month in production cost $70,000. Prompt engineering also became difficult because GPT-4 was not a financial expert. Detailed instructions and examples made prompts long, and fixing one scenario often broke another. The team also lacked prompt versioning.
Production requirements must be measured against the real task
Method expected at least 16 million requests per day, 100,000 concurrent load, and sub-200-millisecond latency. Kyle recommends measuring error rate, latency, and cost under real production conditions, with the same task diversity and concurrency. Method could measure extraction errors by having a human determine the correct financial number and comparing it with the agent's final output. Additional sanity checks tested whether outputs were plausible and matched earlier information. Method needed roughly a 9% error rate after accounting for those checks.
Existing models missed at least one production requirement
The benchmark showed that GPT-4 had about an 11% error rate and responded in around one second for Method's task. o3-mini had about a 4% error rate but took around five seconds and generated more reasoning tokens, which made it slightly more expensive for this use case despite its lower per-token price. Neither model met all three requirements. GPT-4 missed the target on error rate and cost, while o3-mini was too slow and still missed the cost target.
Fine-tuning moved the task to a smaller model with a better price-performance tradeoff
Kyle presents fine-tuning as a power tool to use after prompting and production benchmarking have failed to meet requirements. OpenPipe used Method's production inputs and generated outputs from o3-mini as training data. The resulting 8-billion-parameter Llama 3.1 model performed better than GPT-4 on the task and beat Method's required error threshold. Smaller size also reduced the amount of computation, which lowered latency and cost. The model could potentially run inside Method's infrastructure and avoid network latency.
A working production system can start with existing data and a narrow use case
The speakers say the project did not require buying GPUs or a large engineering team. Method had already collected production data from GPT, chose a specific use case, fine-tuned a relatively small model, and selected the cheapest model that delivered the needed speed. Kyle cautions that fine-tuning takes more engineering work than prompt engineering, but it can be worthwhile when quality, latency, and cost targets cannot be met with available models. Mustafa adds that AI agents require patience because their behavior takes time to become reliable in production.
"We didn't have to go digging around for the data in the first place."17:25
Who should watch
You are running an AI extraction or agent workflow whose prompts work in tests but become expensive or inconsistent in production.
Your application has clear quality, latency, and cost targets and you need a practical way to compare models against them.
You already have production inputs and outputs and want to know when fine-tuning a smaller model is worth the extra engineering work.
