General-purpose models can answer many financial queries, including misspelled or incomplete ones, but they often fail to follow incorrect or irrelevant context.
2
Writer's financial benchmark measures both answer correctness and whether a model follows the supplied context.
3
Waseem Alshikh argues that financial systems still need domain-specific models alongside grounding, guardrails, and other components in a full-stack system.
Summary
Waseem Alshikh presents Writer's financial benchmark for testing language models in realistic finance scenarios. The benchmark includes misspelled, incomplete, and out-of-domain queries, along with missing context, OCR errors, and irrelevant documents. It measures whether a model gives the right answer and whether that answer follows the supplied context. General and reasoning models often perform well on imperfect queries, but the results change when the context is wrong or noisy. Larger thinking models can be worse at grounding, producing answers that ignore the attached material and increase hallucination risk. Alshikh says that even the best models in the evaluation leave a substantial gap between answer accuracy and reliable context following. His conclusion is that financial applications still need domain-specific models, grounding, guardrails, and the surrounding system components required for dependable use.
Writer builds both general and domain-specific model families
Waseem Alshikh introduces Writer's history, starting in 2020 and building Transformer-based models. He says Writer has published around 16 models, with another 20 in development. The models fall into general categories, such as Palmyra, and domain-specific categories for areas including creative work, financial services, and medicine. This leads to the talk's central question: if general models now reach roughly 80% to nearly 90% average accuracy, is it still worth building models for particular industries?
The financial benchmark tests failures that ordinary prompts miss
Writer created a benchmark called FLaME to evaluate models with real-world scenarios. Its query-failure tests include misspellings, incomplete queries, and questions outside the user's domain. Its context-failure tests include missing context, OCR errors from converting physical documents into text, and irrelevant context such as uploading the wrong document. The dataset, evaluation set, leaderboard, and paper are open source through GitHub and Hugging Face. The evaluation measures whether the model gives a correct answer and whether it follows the supplied grounding or context.
Answering a question is easier than following its context
The benchmark results show that nearly every model can produce some kind of answer, with reasoning models sometimes scoring slightly higher on that measure. The larger differences appear in grounding and context-following tasks, including text generation and question answering. When a prompt contains misspellings, incomplete wording, or an out-of-domain request, several models still perform well. That does not mean they will respect the document or context attached to the question.
Reasoning models can perform worse when the context is wrong
Alshikh says the larger thinking models are roughly 50% to 70% worse on grounding in the examples he shows. They answer questions even when the answer is outside the supplied context, which raises hallucination risk. Smaller models perform better in some of these context tests. The result challenges the assumption that more thinking automatically produces more reliable behavior in domain-specific tasks.
Financial reliability requires more than a language model
The benchmark shows a large gap between robustness, hallucination behavior, and getting an answer correct. Alshikh says that even the best models remain below 81% when the evaluation considers robustness and context grounding. In a financial workflow, that would mean a meaningful share of requests could be wrong. He argues that a usable system needs the model plus grounding, guardrails, and the other components around it.
Alshikh returns to the opening question and answers yes. Based on the benchmark results, Writer will continue building domain-specific models even as general-model accuracy improves. The remaining weakness is context following. Models still fail to use the supplied context correctly, and Alshikh says that this gap remains far behind what financial applications require.
"You need the full stack, you need the system, you need the grounding, you need everything from guardrails and the build around the system itself."10:40
Who should watch
You are evaluating LLMs for finance, medicine, customer support, or another domain where a plausible answer can still be wrong.
Your team compares reasoning models with smaller or domain-specific models and needs tests for grounding, OCR noise, and irrelevant documents.
You are designing a production AI system and want evidence for why model choice alone does not solve reliability.
