Finance models need domain-specific training because general models have broad coverage but can lack depth on technical financial information.
2
Gradient built a data pipeline that filters already-seen documents, uses human review, augments the remaining data, and trains with continual pre-training, supervised fine-tuning, and preference optimization.
3
Long context lets financial systems use more examples and source material directly in prompts, which Leo says can reduce hallucinations and simplify execution.
Summary
Leo Pekelis describes Gradient's approach to building finance-focused language models. He argues that general models often lack enough depth on technical financial material, even when they are very large. Gradient's pipeline collects financial documents, estimates whether the base model has already seen them, sends the remaining data through human review, and adds synthetic variations. The resulting training process combines continual pre-training with supervised fine-tuning and preference optimization. Pekelis also explains why long context matters for financial workflows. In-context information can correct gaps in model memory, but prompts quickly become too small for examples, related documents, or complex reasoning. Gradient extended one model to about one million tokens, allowing it to use thousands of examples and more connected source material. The talk presents the finance model and context extension as separate parts of a larger system, with open-source releases and benchmark results offered for inspection.
General-purpose models lack depth on technical financial information
Pekelis says general models such as GPT-style systems are trained on broad data rather than deeply on specialist material. He cites research showing that even a 176-billion-parameter model needs thousands of relevant documents in pre-training to achieve above 50% accuracy on a related question. When information sits at the tail of a model's training distribution, the model answers poorly. Gradient's response is to train a finance-specific model instead of relying on an off-the-shelf base model.
Automated curation makes a large financial corpus reviewable
Financial data is too plentiful for people to inspect manually, so Gradient built an automated pipeline. Its curation process borrows from membership inference research to estimate whether a document was already present in the base model's training data. Gradient filters out documents the model has already seen, leaving a smaller set that people can review. The reviewed data then passes through synthetic augmentation, which adds more examples and variations in representation and formatting.
The training recipe separates learning finance from learning how to use it
Gradient starts with an existing base model, such as a Llama model, and applies continual pre-training with next-token prediction on the curated finance data. It then runs supervised fine-tuning and preference optimization. Pekelis compares pre-training to reading textbooks and alignment to taking an exam or completing a project. The first stage helps the model retain information, while the later stages teach it how to apply that information and follow preferred practices.
In-context information is a direct way to cover gaps in model memory
Pekelis defines hallucinations as generated content that is irrelevant, made up, or inconsistent with the input. He connects them to problems such as outdated training data and errors in automated data collection. Continued training and alignment can help, but Gradient finds that in-context learning is the most direct and sample-efficient response. Putting information into the prompt at inference time can cover weaknesses in the model's stored training data.
Prompt size becomes the bottleneck when examples and documents multiply
In-context learning works well enough that teams soon want to add more material than the prompt can hold. Few-shot workflows may run out of space before they run out of useful examples, forcing example selection or lossy summaries. Complex tasks may need brittle preprocessing pipelines. Retrieval-augmented generation also struggles when retrieved chunks depend on one another, since one chunk may be needed to know which related chunk to retrieve.
A million-token context can move domain adaptation into inference
Gradient extended some models to roughly one million tokens. Pekelis says this makes it possible to place thousands of examples directly in a prompt, creating an adaptive form of domain learning at inference time. He says this scale of examples is necessary for production-level accuracy on many tasks. The model can also attend across a much larger body of material, allowing more complicated reasoning without as many external workflow components.
Long context can support style and content inference from connected source material
For an internal demonstration, Gradient removed identifying information from books by Mark Twain and placed five books into the model's context. The model generated new stories in the author's style. A separate critic model judged that the stories could have been written by Twain, based on more than surface language. Pekelis says the similarities reached themes, characters, and settings. He presents this as an example of the model using attention across a large prompt.
The released models target both finance tasks and long-context retrieval
Gradient open-sourced the V-alpha-tross model, built by applying finance-domain training to a Llama 2-based model. Pekelis says it was competitive on general open LLM benchmarks and better on finance-specific benchmarks than models in its class. Gradient also released a million-token context extension based on Llama 3. It achieved 100% needle-in-a-haystack scores above one million context lengths and improved substantially on NVIDIA's RULER benchmark.
"Pre-training is something like if you had a bunch of textbooks and you wanted a model to read all those textbooks and understand all that information or retain all that information."07:19
Who should watch
You are building a finance agent whose workflow depends on many chained model calls and needs better execution reliability.
Your system depends on retrieval, few-shot examples, or long source documents, and prompt size is forcing lossy summaries or fragile preprocessing.
You are training a domain model and need a concrete recipe that combines data curation, continual pre-training, supervised fine-tuning, and preference optimization.
