When AI is confidently wrong

A language model produces plausible text, not true text. Those two often coincide, and not always.

Why that is dangerous

A confidently phrased error is harder to spot than a hesitant one. That is exactly what makes checking necessary.

Where the risk is highest

Figures, dates, legal references and proper names. Those are what a model most readily invents, because they resemble what it has seen.

What reduces the risk

Supplying the sources and asking it to stay within them. A model answering from a document you give it is wrong far less often than one answering from memory.

Human checking

Put it where an error is expensive. An internal summary needs less verification than an answer sent to a customer.

What to disclose

If your customers interact with an automated system, say so. A forgiven mistake is one that was acknowledged as such.