GPT-OSS-120B forecast Costco's two-year profit increase within 4.4% of the actual result using only filings available before the 2024 membership fee increase.
2
The forecast contained two bad assumptions: it applied the fee increase to the wrong member population and used a 15% executive-membership share instead of the disclosed 45%.
3
A subject-matter expert should review an agent's assumptions and inputs, then turn those accounting checks into code where possible.
Summary
Hassan El Mghari and Omar El Mghari test whether an open model can forecast Costco's profits without seeing information about a later membership fee increase. They convert 214 public filings into about 3,000 timestamped facts, give the pre-cutoff facts to GPT-OSS-120B, and ask it to write structured formulas rather than calculate everything in prose. Sixteen cheap runs produce five models that pass engineering checks. Their average forecast is within 4.4% of Costco's actual result. Omar then audits the assumptions and finds that the apparent accuracy came from two errors cancelling each other out. The model applied the fee increase to all members instead of the US and Canada population, and it guessed a 15% executive-membership share even though filings disclosed 45%. Correcting one assumption increases the error to 28%. The talk argues that automated checks are only one layer. Domain experts need to inspect assumptions, and their checks should become code when possible.
Forecasting requires a model that has not seen the event
The team asks whether an LLM can forecast a company's profits using only public information available before a known event. Costco provides a test because it raised its membership prices in July 2024, from $60 to $65 for the standard tier and from $120 to $130 for the executive tier. Newer models already knew about the increase from their training data, so Hassan used older open models whose training ended before the change. He presents this as a practical requirement for forecasts: the model's training data should end before the event being predicted.
Costco membership fees have an outsized effect on profit
Costco sells about $300 billion of goods each year at close to cost, so its membership fees matter greatly to profitability. Hassan says more than half of Costco's revenue comes from membership fees, while total operating profit was about $12 billion and slightly more than half came from those fees. That makes the 2024 price increase a useful forecasting case. The question is how much extra revenue and profit the higher fees will generate, given the membership tiers and the way Costco recognizes the fees over time.
Structured formulas keep the model's reasoning inspectable
The team scrapes 214 quarterly and yearly filings and turns them into about 3,000 timestamped facts, then filters out anything after the cutoff. GPT-OSS-120B writes a financial model from those facts. Instead of asking the model to do all the arithmetic at once, they have it produce structured formulas and assumptions. One formula captures Costco's deferred treatment of membership revenue: a $65 payment is spread across 52 weeks. The resulting model contains assumptions, 14 formulas, and values taken directly from filings. This structure makes the model easier to run and inspect.
The team runs 16 model sessions in parallel, asks each one to build the financial model, and applies basic checks. A formula cannot depend on itself, and a membership fee increase should raise revenue. When a run fails, its error is sent back to the model for up to two correction attempts. Some runs still fail, so the team keeps the five that pass. Hassan says this approach cost 19 cents and was more effective than repeatedly tuning one prompt. The pattern works because the model is cheap enough to run many times.
The forecast's accuracy came from two errors cancelling out
Omar audits the assumptions instead of accepting the 4.4% result. He finds that the model used a 15% executive-membership share based on a citation that supported increased penetration but did not provide that number. Costco's filings already disclosed a 45% figure, which the model missed. The model also applied the price increase to the entire member population instead of only the US and Canada portion. The first error overstated the base and the second understated the executive share. Together they produced a forecast that looked close. Replacing 15% with 45% alone raises the error to 28%.
Engineering checks need accounting checks and human review
Omar describes three lines of defense. The first is management control, which corresponds to Hassan's engineering checks. The second is an internal audit or accounting review of assumptions and model behavior. The third is an independent external check. The model passed the first layer but did not catch that a cited assumption had no supporting number or that a disclosed 45% figure had been missed. Hassan says a subject-matter expert should inspect the agent, its inputs, assumptions, and outputs. Their domain-specific checks can then be encoded in the model to reduce the amount of manual review.
"So we had this LLM build a financial model. We evaled it. Ended up giving us this result. And then I showed it to my dad and it turned out it got the right answer or got close to the right answer, but for the wrong reasons."Hassan El Mghari00:32
Who should watch
You are building an agent for finance, accounting, forecasting, or another field where assumptions matter as much as outputs.
You want to evaluate a model against an event without letting its training data reveal the answer.
You are deciding which expert reviews should become automated checks inside an agent.