Earnings prediction is a better financial-agent test than stock prediction because it has numerical ground truth, analyst consensus as a baseline, and explanations for surprises.
2
Samaya AI found that the latest frontier models were starting to outperform analyst consensus across more than 450 earnings reports.
3
Live, point-in-time data mattered more than the model's stored knowledge, while agents often failed when they found useful evidence but did not act on it.
Summary
Ashwin Paranjape argues that stock prediction is too noisy to make a good first test for financial agents. It is hard to know why a stock moved or whether a model's reasoning was correct. Earnings are easier to evaluate because companies report specific numbers, analysts publish consensus estimates, and earnings calls explain surprises. Samaya AI built a point-in-time environment that prevents an agent from seeing information released after its prediction date. Across more than 450 earnings reports and seven frontier models, the newest models began to beat consensus on several measures. Agents did well when they found timely evidence and moved decisively away from consensus. They missed when they noticed evidence but lacked the conviction to use it. Paranjape's strongest result concerns data access: a model with only its pretraining knowledge performed poorly, and stale data caused a large drop. The talk ends with work on expert guidance, hidden-consensus tests, internal customer data, reasoning, and continual learning.
Earnings prediction gives agents measurable ground truth
Earnings provide a concrete task every quarter. Companies report revenue, gross margin, operating income, EPS, and other metrics, then provide guidance for the next quarter. Analysts publish estimates that are pooled into consensus. When the company reports, the prediction can be compared with the actual number and classified as a beat or miss. The earnings call and later commentary also explain why the surprise happened. Paranjape says this gives the evaluation both quantitative ground truth and qualitative information about whether the reasoning was sound.
Point-in-time access prevents agents from seeing the answer
Samaya AI built an agent environment around a production harness with company documents, prices, consensus estimates, and news. A programmatic point-in-time gate limits every source to information available at the prediction moment, such as five sessions or seven days before earnings. The gate uses an authentication token and applies to document retrieval and structured-data tools. Web access is harder to control, so Samaya uses timestamped news streams as a proxy. The agent researches inside this environment, makes its prediction, and is evaluated when earnings are released.
The evaluation covered more than 450 earnings reports and seven frontier models. Consensus had an error of 1.5 times historical volatility, while a bias-corrected consensus measure had an error of 1.18. Paranjape says the newer models reduced error as model capability increased. On revenue, GPT-6 Astra was significantly better than the bias-corrected consensus in the comparison he presented. Newer models also improved hit rates, including calls below consensus, which he described as harder than the usual majority baseline.
Agents performed well when they gathered specific evidence and adjusted away from consensus. In the Macom example, the agent found information from a small UK wafer supplier that revealed improving data-center demand. It followed that signal through to the profit-and-loss statement and predicted a result much higher than consensus, close to the actual outcome. Paranjape describes this as a strength of agents: they can search across a large amount of current information and use a relevant detail to revise a forecast.
The agents also failed after finding evidence that should have changed the prediction. For Ross Stores, an agent found that a peer had booked a tariff refund. It chose to leave that information out of its gross-margin forecast, so its prediction stayed close to consensus while the actual result came in much higher. Paranjape says these misses happen when the model sees the signal but does not have enough conviction to use it. The size of the surprise also mattered: models tended to do better when outcomes were far above or below consensus.
Live data matters more than stored model knowledge
Paranjape compares agents with no data access against agents that receive stale information. A model with only its parametric knowledge performed poorly, even though its training included documents released before the cutoff. Giving it data from 11 weeks before the earnings call helped, but performance remained 12 to 15 points lower than with current data. He says that gap was similar to the quality difference between the weaker and stronger models in the test. The result suggests that current information, rather than model memory alone, drives the forecasts.
Expert guidance increases research and can improve error
Samaya also tested expert guidance. Without it, agents used fewer tokens and spent fewer minutes researching. With guidance, they did more research, and Astra showed a significant improvement in predicted error. The team separately removed every accessible mention of consensus, including references extracted from news. The strongest models reconstructed the consensus view themselves, with Astra showing no significant difference between having consensus and not having it. Older models benefited much more from direct consensus access.
The next tests involve customer data and continual learning
Samaya has made the system available for alpha testing and is working with customers to test whether their internal data improves predictive tasks. Paranjape also wants to understand whether model reasoning resembles human reasoning and whether the two can complement each other. Another direction is post-training and continual learning, where agents learn from mistakes made on earlier predictions. He also points viewers to Samaya's blog post and Frontier Finance for more detail on evaluating financial harnesses.
"This is a great task. It has quantitative ground truth, an expert human baseline to compare against, and qualitative ground truth that tells you why the agent was right if it was right."03:56
Who should watch
You are building or evaluating financial research agents and need a task with a clear outcome, a human baseline, and explanations for errors.
You want to test whether an agent can use current company and market information instead of relying on what was stored during training.
You are deciding how much point-in-time data control, expert guidance, or consensus information to expose to a financial model.