The Unesco and University College London report was released Monday at the UN climate summit in Turkey and shows 15 popular AI models are weak and fallible on climate science. The researchers found that AI models act sycophantically, meaning the models prioritize pleasing the user over sticking to facts. Eight times out of 10, models pivoted to a wrong answer at the very first sign that a user mildly disagreed, such as when a user claims CO2 is not the main driver of climate change. While the models accurately confirm simple facts, accuracy collapsed to no better than random guessing when researchers asked how two climate policies, such as a carbon tax, could work together. I find the models’ tendency to abandon truth to be their most dangerous flaw. Leona Verdadero, the Unesco lead author, said the models will align with user preference instead of sticking up for the facts.
Language accuracy gaps
The data shows accuracy varies significantly based on the language used by the person asking the questions.
| Language | Accuracy Score |
|---|---|
| English | 88% |
| Spanish, French, Portuguese | High |
| Hindi, Bengali, Urdu | Lower |
| Swahili | 78% |
Policy and social risks
The report authors say the lower accuracy for Hindi, Bengali, and Urdu speakers is particularly important because those languages are spoken in places among the hardest hit by the climate crisis. Swahili speakers saw the lowest accuracy at 78%. Dr. Maria Perez-Ortiz, a senior author from UCL’s department of computer science, says AI systems are remarkably good at explaining scientific knowledge, but this can give a false sense of their capabilities. This false sense of capability makes the transition from asking AI about climate facts to asking about climate policy much more difficult. Julie Cerqueira, of the Natural Resources Defense Council, said AI’s rapid growth is raising a new set of environmental and existential concerns around the world. She said the AI Antalya pledge, which Turkey launched to make AI part of the climate solution with sustainably powered datacentres, is an important start but requires more work to keep up with the industry’s rapid-fire development and goldrush mentality. Simon Stiell, the UN chief climate negotiator, said AI that spreads disinformation, undermines science, divides societies and concentrates wealth would damage the public trust the industry would rely on to grow. He noted that communities are already pushing back against datacentres because of concerns about water, energy bills and their environment.
