AI Chatbots Get Most Money Questions Wrong in New Test
Popular AI chatbots gave wrong answers to financial questions 57% of the time, according to research covered by the Financial Times on Sept. 19.
The finding gives a plain starting point for general-purpose assistants on money topics. It covers mainstream systems, not an unusual edge case.
What the test found
The test used more than 100 money-related questions. It covered models from ChatGPT, Claude, Copilot, Grok and Gemini. Financial Times Coverage across five model families points to a system-wide limit in handling regulated topics where rules change, rather than a bug in one product.
Error rates rose sharply on harder items. Some models gave wrong answers to 99% of harder financial questions, according to the Sept. 19 account. Financial Times A Sept. 15 account had put errors in advanced finance-related responses at 88%. IBS Intelligence The later figure is the firm read for the hardest set. Both numbers leave little doubt. Difficulty raises the risk.
Why dates and rules trip chatbots up
One failure was about timing. Some chatbots ignored upcoming tax changes when answering financial queries. Financial Times Tax rules carry a version and a start date, much like a software update. A model can retrieve the old rule and state it fluently, so it sounds right while being wrong. Engineers will recognize the parallel to a stale cache, where stored data is served after it should have expired. Getting it right depends on effective date, not only on finding a related rule.
Regulators have tracked this area for some time. The U.S. Consumer Financial Protection Bureau published a report titled "Chatbots in consumer finance" on chatbot use by financial institutions and associated challenges on June 6, 2023. Consumer Financial Protection Bureau More recently, the Financial Stability Board issued a consultation report titled "Sound Practices for Financial Institutions' Responsible AI" seeking feedback on proposed sound practices on June 10, 2026. Neither document is the source of the 57% figure. They define the governance background in which that figure will be read.
The broader context here is what happens when a system built to predict likely text meets topics that need exact answers. General chatbots are tuned to produce plausible continuation under uncertainty, and their sense of confidence, what specialists call calibration, stays weak in specialist areas. Finance punishes that design. Interest calculations, eligibility thresholds, allowances and filing deadlines do not allow approximate recall. A confident restatement of an outdated rule is worse than a refusal, because the user often cannot tell the two apart.
What would have to change
Looking at what this means for builders, the task splits into grounding, versioning and abstention. Grounding ties an answer to a cited, current source rather than the model's built-in memory, what specialists call parametric memory. Versioning ties that source to place and effective date, which is where the missed tax changes bite. Abstention matters most. A system that declines or hands off to a person when confidence is low, or when rules are in transition, will score lower on helpfulness tests and do better where legal responsibility sits. For tech-literate users the practical lesson is similar. Treat the chatbot as a drafting aid for concepts, not as a calculation engine or compliance guide.
In my view, the result is not a reason to dismiss language models in finance. I have watched my own children move from search boxes to conversational answers for everything from homework to bank fees, and the pull of fast plain-language help is clear. The technology is useful for summarization, sorting requests, pulling facts from documents and writing first drafts. The gap is between explaining a concept and standing behind an outcome. Closing that gap will need narrower uses, tighter links to tools for math and dates, test sets built around law changes rather than fixed trivia, and logs that record which rule version produced which answer. That work is unglamorous. It is also how enterprise search, payments infrastructure and cloud controls matured, through iteration driven by measured failure.
Worth flagging is what success would open up. Reliable assistants that track rule changes, cite provisions and refuse cleanly could expand access to basic financial guidance at very low extra cost. The current numbers show we are not there with open-ended chatbots. The path forward will look less capable and be more trustworthy.
