AI Can Be Confidently Wrong. Accountants Cannot.

By LaMichelle Hecht, CPA

Artificial intelligence can save time and strengthen research. But when an answer sounds polished and authoritative, that is exactly when accountants need to slow down and verify it.

Artificial intelligence can draft an email, summarize a long document, and create a checklist before most of us finish our first cup of coffee. That speed is exciting. It can also be dangerous.

Generative AI sometimes produces information that sounds reasonable but is inaccurate, unsupported, or entirely made up. These errors are commonly called "hallucinations." The National Institute of Standards and Technology calls this phenomenon "confabulation," referring to AI-generated content that is confidently presented but erroneous or false. OpenAI describes hallucinations as plausible but false statements and acknowledges that they remain a challenge even as models improve.

For accounting professionals, this is not a cute technology glitch. A fabricated IRS citation, outdated accounting standard, or nonexistent audit requirement can find its way into a client memo, tax position, grant report or workpaper. Once it is wrapped in professional language, the error can be difficult to spot.

Let's be honest: AI is very good at sounding like it knows what it is talking about. Confidence is not evidence.

Use AI as an assistant, not the signer.

I use AI regularly, and accounting professionals should learn how to use it too. It can organize our thoughts, create a first draft, identify questions we missed, and turn a messy process into a usable checklist. It can make a good employee more efficient and give a growing firm more capacity.

But AI should be treated like an eager assistant, not the final authority. It does not know your client the way your team does. It does not carry your license or take responsibility for the conclusion.

Guidance from the American Institute of CPAs and the Chartered Institute of Management Accountants makes the same point: AI can improve efficiency, but it does not replace professional judgment.

Firms also should not view AI as a reason to eliminate the people who develop that judgment. Staff need opportunities to research, receive coaching, and learn how to reach sound conclusions. As AICPA & CIMA has noted in discussing AI's impact on the accounting talent pipeline, preserving mentorship, career growth and professional judgment remains important as firms pursue greater efficiency.

If AI performs every entry-level task and no one teaches the next generation how to think, we may save time today and create a talent problem tomorrow.

The goal is not artificial intelligence instead of staff. It is staff using artificial intelligence well.

Do not ask only for sources. Ask for receipts.

Many people tell AI, "Cite your sources," and stop there. That is not enough. Generative AI can produce citations and references that appear legitimate but are inaccurate or nonexistent. It may also cite a legitimate source that does not support its statement.

Accountants need a source, a precise location and independent confirmation.

Ask the AI to identify the authoritative source for each material claim, including the exact section, paragraph, page or other pinpoint reference. Then open the source yourself. Confirm it exists, applies to the correct period and jurisdiction, and supports the conclusion in context. Depending on the assignment, check the applicable code, regulation, professional standard, contract, award document or court authority.

Also instruct the AI not to guess with the following command:

Do not invent a citation. If you cannot verify the source, say that clearly.

AI systems may be more likely to guess when uncertainty is penalized, so it helps to explicitly allow the tool to acknowledge when it does not know. In accounting, honest limitation is more useful than confident fiction.

The prompt matters.

Consider this prompt:

Write a memo explaining why our nonprofit client can recognize a $500,000 government grant as revenue when the award is signed. Cite the accounting guidance and make the position persuasive.

This prompt is more likely to produce an unreliable answer because it starts with the desired conclusion. It provides almost none of the facts needed to evaluate the transaction and asks the AI to make the answer persuasive. The tool may fill in missing information, ignore facts pointing in another direction, or manufacture support for the conclusion it was told to defend.

A stronger prompt would be:

You are assisting a CPA with preliminary research, not making the final accounting conclusion. A nonprofit received a $500,000 government award. The agreement was signed June 30. Funds will be paid as qualifying program costs are incurred; unused funds must be returned, and the award includes measurable service requirements. Identify the accounting questions that must be resolved before determining when revenue should be recognized. Consider more than one possible treatment. For every material statement, provide the authoritative source and exact topic, section and paragraph or other pinpoint location. Do not invent citations. If you cannot verify a source, say so. Separate known facts, assumptions, analysis, missing information and next steps. End with a verification checklist for the CPA.

This version gives context, asks for alternatives, exposes assumptions and makes verification part of the assignment. It still does not make the answer correct. It gives you a better starting point and makes unsupported reasoning easier to find.

Correct the tool, then document the lesson.

When AI gets something wrong, do not simply close the chat and try again. Tell it exactly what was incorrect. Provide the correct source or fact, ask how the error affected the answer and require a revision.

For example:

Your citation to Section X is incorrect. The correct guidance is Section Y, paragraph Z. Revise the analysis, identify every conclusion affected by the error, and do not reuse the incorrect citation. Repeat the correction back to me before revising.

Correcting an error can improve the current interaction, but it should not be treated as a permanent control. Some AI platforms offer memory or project-level instructions that may carry information into future interactions, but those features should not replace a firm's own controls.

For recurring work, document important corrections in an approved prompt library, written firm guidance or a review checklist. Those controls are more dependable than assuming an AI system will permanently retain a correction or never repeat a mistake.

AI can help us move faster, but speed is not accuracy. Professional skepticism did not retire when generative AI arrived. Technology can prepare a first draft, organize research and point us toward questions. The accountant must verify the authority, evaluate the facts and own the conclusion.

"The chatbot told me so" will not hold up in a file review, audit inspection or conversation with a client. Nor should it.