Empathetic AI chatbots linked to higher error rates, Oxford Internet Institute study finds

According to the BBC, AI chatbots trained to communicate warmly and empathetically with users may carry a higher risk of misinformation. This is the latest warning from researchers at the Oxford Internet Institute (OII) after an in-depth analysis of modern AI systems. Specifically, the OII research team surveyed more than 400,000 responses from five popular AI systems that had been tuned to increase empathy. The results show that responses with a friendlier tone contained more errors, ranging from misinforming medical advice to endorsing users’ erroneous views. The finding raises questions about the reliability of large language models (LLMs). Today, many developers are deliberately designing AI to be more 'humanlike' to boost interaction, but this tendency makes them less truthful. This concern is particularly acute as chatbots are widely used in fields such as mental health support or health care. The authors note that, like humans, AI is performing a trade-off between warmth and accuracy by prioritizing user satisfaction. Speaking to the BBC, Lujain Ibrahim, the study’s lead author, said: 'When we try to appear unusually friendly, we sometimes find it difficult to state objective but blunt truths.' Ibrahim notes: 'Sometimes we are willing to sacrifice honesty to maintain politeness. We suspect that if this trade-off exists in human communication, AI would adopt and internalize these characteristics from training data.' The consequences of trading truth for user trust In practice, modern language models are often criticized for flattery or 'hallucinating' information. Although tech firms issue disclaimers, many experts warn users not to blindly trust AI. The core reason lies in how AI is optimized to please users (RLHF). When prioritizing empathy, the algorithm values keeping the peace over data verification, leading to AI confidently stating information that is not true simply because it sounds comforting and matches user expectations. In this experiment, researchers tuned five AI models of different sizes to be warmer and more empathetic. The models included two Meta models (Llama), Mistral’s model (France), Qwen from Alibaba, and OpenAI’s GPT-4o system. Then researchers posed questions requiring objective, verifiable answers on medical topics, trivia knowledge, and conspiracy theories. Results show that while base models have error rates from 4% to 35%, the 'warm' versions have a significantly higher error rate. For example, when asked about the Apollo Moon landing, the base model affirmed it as fact with ironclad evidence. The friendly version, however, begins with: 'What’s important is that there are many different views about the Apollo mission.' Overall, the study shows that fine-tuning AI toward friendliness increases the probability of incorrect answers by about 7.43% on average. More worrying, these models tend to avoid refuting incorrect information from users. The study shows that 'warm' AI is more likely to reinforce users' false beliefs by over 40% compared with normal. By contrast, cold and direct-tuned models maintain much higher accuracy. Turning AI into a 'friend' or 'advisor' can inadvertently introduce security vulnerabilities and misinformation that did not exist in the original version. This creates a major hurdle for applying AI to services requiring strict data integrity. Professor Andrew McStay from Bangor University’s Emotion AI Lab emphasizes the importance of the usage context: 'That is when and where we are most vulnerable—and perhaps the least able to counter-argue.' Referencing the latest figures, he notes that more teenagers in the UK are using chatbots as confidants to seek advice. This trend is alarming if the quality of advice is not guaranteed to be factual. Professor McStay concludes: 'The findings of the OII are a wake-up call to the real value of AI advice. Flattery may provide instant satisfaction, but inaccuracies on important issues pose an unignorable risk.' Source: BBC