His accountant found the $14,000 error three weeks late

A retiree asked a chatbot whether converting part of his 401(k) to a Roth IRA would push him into a higher tax bracket. The answer came back clean, formatted, and confident. It was also off by about $14,000 in projected tax. He found out from his accountant three weeks later, after he had already started the conversion.

When researchers feed chatbots batteries of real financial questions, tax treatment, loan amortization, benefit eligibility, withdrawal rules, the share of answers with a material error runs past half. “Most of the time” is not a headline stretch. On the specific, number-dependent questions people bring to these tools, a wrong answer is the base case.

The reason is structural. Once you see how the machine works, the error rate stops looking like a bug and starts looking like the expected output.

A language model predicts words, not balances

A chatbot generates the most statistically likely next word given everything before it. That is the whole job. It has no ledger, no tax table it consults, no running arithmetic. When it writes “your effective rate would be about 22 percent,” it is producing text that resembles correct financial writing, not reading a number off a calculation.

Sometimes the likely-sounding words are also the true words. On a question like “what is a Roth IRA,” the training data is thick with correct explanations, so the odds favor a right answer. On a question like “if I withdraw $40,000 from a traditional IRA at 63 while collecting Social Security in Ohio, what is my marginal rate,” the answer depends on this year’s brackets, provisional-income thresholds, and state rules interacting. The model has seen the shape of that answer far more often than the answer itself. It fills the shape.

This is why the failures cluster on exactly the questions that matter. The higher the stakes and the more specific the numbers, the more the model is guessing.

Four failure modes you can spot without a finance degree

Stale numbers. Contribution limits, standard deductions, and interest rate assumptions change every year. A model trained on data through one year will state last year’s figure with full confidence. It has no clock and does not know its own numbers expired.

Invented rules. Ask about an obscure provision and the model will often produce a rule that sounds plausible and does not exist. A financial regulation that is 90 percent right is 100 percent wrong when you file. There is no “close enough” on a tax return.

Arithmetic that does not add up. Language models are poor at multi-step math. Amortization, compounding, and tax-bracket stacking all require carrying numbers across several operations. The model can narrate the steps correctly and still land on a total that is off, because it is describing arithmetic rather than doing it.

Jurisdiction blending. Ask a general question and you may get an answer that mixes U.S. federal rules, a state rule, and something that only applies in the U.K. The model has no reliable sense of which legal system your question lives in unless you force the boundary and even then it drifts.

None of these require an expert to catch in hindsight. They require someone to check before the money moves.

The confidence is the dangerous part

A human advisor who does not know an answer hedges. They say “I would need to check the current limit” or “that depends on your state.” The uncertainty shows in their voice, and you calibrate.

A chatbot strips that signal out. It delivers a hallucinated rule in the same measured, well-organized tone it uses for a rule it has right. There are no error bars. The formatting, the bolded headers, the tidy numbered steps, all of it reads as authority, and none of it correlates with accuracy. You are looking at a system that is equally fluent when correct and when fabricating, which means fluency tells you nothing.

That is the specific trap in finance. In casual use, a confident wrong answer about a movie plot costs nothing. A confident wrong answer about a required minimum distribution triggers a penalty of 25 percent of the amount you failed to withdraw. The tool’s tone is identical in both cases.

What human oversight actually looks like

Oversight is not “read it and see if it sounds right.” Sounding right is the failure mode. Oversight means a specific person applies specific checks before a decision executes. Four that work:

Verify every number against a primary source. If the chatbot cites a 2026 contribution limit, open the IRS page and confirm it. Treat the model as a search prompt that points you toward what to look up, not as the source itself. The moment you copy a figure straight into a decision, you have removed the only step that catches the stale-number failure.

Make it show the math, then redo the math. Ask for each step, then run the calculation yourself or in a spreadsheet. If the model claims a monthly payment, plug the loan terms into an amortization formula. The narration and the arithmetic fail independently, so checking the total catches errors the explanation hides.

Name the jurisdiction and the date in the question. “Federal U.S. rules as of 2026” cuts off a whole class of blended answers. It does not fix the problem, but it narrows the surface where the model can wander.

Route anything binding through a licensed human. For a tax filing, a mortgage, an estate decision, or a retirement withdrawal schedule, the chatbot output is a draft for a CPA, an enrolled agent, or a fee-only planner to check. A professional carries liability and a duty of care. The chatbot carries a disclaimer that tells you, in the terms of service, not to rely on it for financial advice.

That last point is worth sitting with. The companies building these tools already tell you not to trust them for this. The oversight is not optional caution layered on top. It is the operating instruction printed on the box.

”But it can search the web now”

The newer tools retrieve live pages and cite them, and this does help with the stale-number problem. It does not close the gap, for two reasons.

First, the model still decides which retrieved sentence to trust and how to combine it with the rest of the answer. Give it a search result for federal brackets and a blog post with an outdated figure, and it can pull the wrong one into an otherwise correct explanation. Retrieval widens what the model can reach. It does not fix the model’s judgment about what it reached.

Second, citation is not verification. The tool will attach a link that looks authoritative and then summarize it incorrectly, because summarizing is the same predict-the-next-word process that produced the error in the first place. People see a footnote and stop checking, which makes a cited wrong answer more dangerous than an uncited one. The footnote raises trust without raising accuracy. Open the link and read the source yourself, or the citation is doing the opposite of its job.

Where the tools genuinely help

The honest read is not that chatbots are useless for money. They are strong at the parts that do not require a correct number.

They explain concepts well. “What is the difference between a traditional and Roth IRA” gets a solid answer, because that content saturates the training data and does not depend on this year’s figures. They draft the boring paperwork, a letter to dispute a charge or a list of questions to bring to an advisor. They translate jargon, turning a dense benefits document into plain language you can then verify.

The line is clean. Use the tool to understand your options and to prepare. Do not use it to decide, and never let it produce the final number. The concept is where the model is strong. The number is where it guesses, and the number is what costs you.

The part that scales badly

One retiree with a bad Roth conversion is a story. The structural risk is that these tools now sit inside banking apps, budgeting products, and customer service chat windows, presented as features rather than as a text predictor with a known error rate. A person who would never trust a stranger’s tax advice will accept the same guidance from a chatbot embedded in an app they already use, because the interface reads as institutional.

The error rate does not drop when you put a bank logo on it. The same model that fails half the specific financial questions in a lab fails them inside the app too. What changes is that the user trusts it more and checks it less, which is the opposite of what the error rate demands.

Treat any financial answer from a chatbot as a confident draft from a system that is wrong more often than it is right on exactly these questions. Verify the numbers, redo the math, and put a licensed human between the output and any decision that moves money. The tool is a research assistant with a serious accuracy problem, and the oversight is the only part of the process that is actually accountable to you.