How AI hallucinations can impact your business

AI is commonplace in business now, with 66 percent of people using it regularly, according to a study of 48,000 people across 47 countries by KPMG and the University of Melbourne.

However, only 46 percent of those people trust AI systems. In fact, 66 percent of people rely on AI output at work without evaluating its accuracy and 56 percent of people said they were making mistakes due to AI.

There’s a good chance that is happening at your company too, but why is trust in AI so low? It comes down to something called AI hallucinations and they’ve affected companies big and small. Below, we’ll go over some real-world examples of how hallucinations have cost real companies money, how hallucinations happen and what you can do about it.

How AI hallucinations have cost real companies

Hallucinations have affected some of the biggest companies in the world. The most famous example is that of Air Canada.

British Columbia resident Jake Moffat was booking a flight for his grandmother’s funeral on Air Canada’s website. A chatbot told him he could be refunded a portion of his ticket cost if he applied for the discount within 90 days, according to ComputerWorld. When he tried, he was told by Air Canada that they wouldn’t honor the chatbot’s claim.

Air Canada was found liable for its AI chatbot and the hallucination and had to reimburse Moffat $600 in bereavement refunds and tribunal costs, half of what he paid for the tickets.

Deloitte Australia used AI to create a welfare compliance review for the Australian government. However, the report contained AI-hallucinated quotes and figures, according to TechSpot. The fabrications were found by a University of Sydney academic.

Deloitte had to pay back a portion of the $440,000 fee for the review to the Australian government.

It’s not just big companies either, Stanford CyberLaw found that 114 cases in the U.S. court system included filings with inaccuracies either caused by or likely caused by AI. Around 90 percent of the law firms involved were either solo practices or small firms. Hallucinations were more common on the plaintiff’s side, 56 percent, rather than the defense, 31 percent.

These hallucinations are costing lawyers, with one California lawyer receiving a $10,000 fine, according to 404Media. Beyond that, these stories tend to go viral and may cause reputational damage to these firms. Would you hire a lawyer who got caught using AI and it cost them their case? You would likely question the quality of their work, like their attention to detail and seriousness.

The business effects of hallucinations are real, but how do they happen in the first place?

How AI hallucinations happen

Artificial intelligence is supposed to be intelligent, right? So how could it just make up things? Well, it’s down to how AI is designed.

The first thing to realize is that AI doesn’t work like human intelligence. It doesn’t necessarily have a brain that stores facts and figures that it can then retrieve. According to MIT Technology Review, the Large Language Models that power AI are probabilistic models.

That means that they recognize patterns and then use those patterns to guess the next word in the sequence. Because of the nature of probabilities, there’s always an inherent chance that AI estimates the next words wrongly.

Plus, problems with training data, incorrect context and unsupported inference can also affect how hallucinations happen.

On top of that, OpenAI says AI hallucinates because standard training and evaluation models reward guessing over uncertainty. That is because when models are tested on accuracy, those tests encourage guessing.

Think about how a multiple-choice test works: you can either get a question right or wrong. Guessing gives you a chance of being right, which can improve your accuracy. Not answering, the equivalent of saying “I don’t know”, always results in zero points. That is the logic that’s incentivized for AI. If it guesses, it has a chance of being right. If it says it doesn’t know, there is no chance of being right.

What you can do about AI hallucinations

Is there nothing you can do about hallucinations? Well, not exactly. There are a few things you can do to help reduce the number of hallucinations you get.

Anthropic published a resource doc with some basic recommendations:

Allow the LLM to say “I don’t know”. Give it permission to say that it doesn’t know something. While this may not always work, it will help.

Ask it to extract “word-for-word” quotes. When you give your LLM large documents, ask it to first extract word-for-word chunks from the text, which Anthropic says can help ground the LLM and reduce false information.

Verify with citations. Ask your LLM to cite its sources for each of its claims. Similarly, you can tell it to verify each claim by finding a supporting quote for it. If it can’t find a supporting claim it must then retract the claim.

Another thing you can do is to push your AI. This, however, means that you will have to act as the verifying source. If something sounds too good to be true or even if it’s something that you didn’t know, ask questions about it. Usually when you tell an AI it’s wrong or ask more questions, it tends to admit where it went wrong — in our experience.

You probably won’t fully be able to escape hallucinations. Sometimes, even with our best efforts, AI can make mistakes. It’s still important to independently verify any information AI provides. While it takes a little longer, it’s much more important to double check and be right than being wrong and having it cost your business down the line.

The wild world of AI

It’s also important to remember that AI is an ever-evolving world. New models are coming at a fast pace, so be kind to yourself and keep in mind that we’re all learning. Keeping up with the latest AI tips and keeping a human in the loop to verify and direct AI are some of the most critical lessons you can take away.