Wells Fargo pre-trained a transformer from scratch on billions of customer transactions, with the model learning to predict the next transaction or token.
2
The difficult engineering work was turning amounts, categories, timing, merchants, postal codes, and other tabular fields into a manageable transaction vocabulary.
3
For fraud detection, the transaction model flagged more fraudulent activity while reducing false positives compared with the production XGBoost model, and it met the latency needed for card approvals.
Summary
Freddy Lecue describes Wells Fargo's transaction foundation model, trained from scratch on billions of debit card, credit card, check, and other customer transactions. The model learns transaction relationships by predicting the next transaction or token. Its learned representation can then be fine-tuned for fraud detection, credit line increases, anti-money laundering, and marketing. Lecue says the hardest part was tokenization. Transaction data contains numerical, categorical, and qualitative fields, so the team had to decide how to encode amounts, time gaps, channels, IP addresses, postal codes, and other attributes without creating an unwieldy vocabulary. The team built a decoder-only model at about one billion parameters and trained it with standard distributed techniques. In fraud classification, attention over a customer's transaction history helped the model outperform the production XGBoost system on detection rate and false-positive rate. The system also processed predictions within the under-250-millisecond requirement for card decisions.
A transaction model can support several banking tasks
Freddy Lecue says a foundation model does not have to be based on language. Wells Fargo has large amounts of structured customer data from debit cards, credit cards, digital checks, and other channels. The team wanted one model that could learn transaction representations at scale and then be fine-tuned for separate applications. He names credit scoring, fraud detection, anti-money laundering, marketing, and credit line increases as possible downstream tasks. This changes the setup from many separately built models to one reusable backbone. The pre-training objective is transaction-based, while the later task determines what the model predicts in production.
The model was trained from scratch rather than adapted from language weights
Lecue stresses that Wells Fargo pre-trained transformer blocks from scratch. The team used the architecture of existing models, but started with zero weights instead of taking an open, pre-trained language model. Training used the bank's transaction data and a GPU cluster. The pre-training task was to predict the next token or next transaction, which helps the model learn patterns across a customer's transactions over time. That prediction task is an intermediate step. The useful output is the learned representation, which can later be post-trained for a particular banking problem.
Transaction data has several dimensions that language models do not provide directly
The raw data covers who is involved, what was bought or transacted, how the transaction happened, when it happened, and where it happened. Lecue says these dimensions expand into hundreds of possible features. The team chose roughly 100 raw features after starting with thousands, then allowed the transformer to learn relationships among them instead of relying on extensive task-specific feature engineering. The vocabulary is specific to financial services, which was one reason the team did not simply reuse an existing pre-trained model. The model had to represent the bank's own transaction types and characteristics.
Tokenization was the hardest part of building the model
Every transaction begins as tabular data, but the transformer needs tokens, IDs, and embeddings. The team had to decide how to encode numerical, qualitative, and categorical values. Examples include transaction amount, the time since the previous transaction, IP address, and postal code. Some fields have only a few categories, while others have many possible values. Lecue gives postal codes as an example where grouping by distributions can produce a more useful representation than assigning a separate token to every code. The team also had to keep the vocabulary small enough for training to remain practical. He calls tokenization the most complex part of the work.
The first useful model had about one billion parameters
Wells Fargo used Chinchilla-related mathematics to guide the architecture and training size, then ran experiments to determine what worked in tabular transaction data. Lecue says the team was not trying to build a trillion-parameter language model. It focused on a transaction model, and the one-billion-parameter version produced the strongest boost among the versions tested while remaining the smallest model that delivered that result. A larger, 11-billion-parameter version was in progress. The architecture was decoder-only because the pre-training task was to predict the next token or transaction.
Standard large-model training methods were sufficient after the data representation was solved
Lecue describes the training setup as conventional once the tokenization and model design were in place. The team used mixed precision, distributed data parallel training, large micro-batches, flash attention, activation checkpointing, and memory-efficiency methods. It evaluated perplexity, base accuracy, and downstream-task accuracy. He says the main questions were how many transactions to place in the context or history and how to adapt the pre-training output for fine-tuning. The model initially took weeks to train, while later training could be done in a couple of days.
Attention over transaction history improved fraud classification
The fraud task asks whether a current transaction is fraudulent given the customer's historical sequence. Lecue says the production XGBoost model was trained on large amounts of fraudulent and non-fraudulent transaction data, but historical relationships had to be manually encoded to make them available to the model. The transformer can attend from the current transaction to earlier transactions directly. Wells Fargo's transaction model flagged more fraudulent activity and reduced false positives compared with the XGBoost system. False declines create calls, operational costs, and lost transactions, so reducing them matters alongside finding more fraud.
The fraud model met the response time required for card decisions
Card approvals have to happen within a 250-millisecond processing window. Lecue says the system was initially about ten times slower from a latency perspective, then reached a level that was acceptable for this requirement. The result was a model that could use a customer's historical transaction sequence without making the approval process too slow. He also connects the detection and false-positive metrics to direct financial effects, since wrongly declining a legitimate transaction can create service costs and lose the transaction. The same pre-trained backbone was being investigated for credit line increases, anti-money laundering, and marketing.
"If you use transformer architectures, you do have this nice attention from transactions to past transactions."14:22
Who should watch
You are building fraud, credit, or anti-money-laundering models from long customer histories and want to understand where a transaction transformer can add information beyond manually engineered features.
Your data is tabular and domain-specific, and you need practical guidance on choosing tokens, vocabulary size, context, and model scale before attempting pre-training.
You need fraud inference to fit inside a strict card-approval latency budget and want to compare detection gains with the cost of false positives.