Technology

UNESCO study finds LLMs struggle with climate policy accuracy

Researchers found that 33 tested models show massive performance gaps in climate policy accuracy, with some falling below 40 percent. Models also lose 30 to 35 percent accuracy when faced with sustained user disagreement and counter-arguments during debates.

UNESCO study finds LLMs struggle with climate policy accuracy

Accuracy and policy predictions

I found that the 33 models tested by Drobnjak, et al./UNESCO show a massive performance gap, where the best models reach 70 percent accuracy and the worst fall below 40 percent. Frontier models score better than earlier generations, and larger open-weight models outperform small ones. The researchers prompted models to evaluate whether evidence supported or refuted specific claims, such as whether the statement "renewable energy investment kills jobs" is accurate, given evidence like descriptions of employment and investment statistics. For 656 questions regarding policy impacts on global temperatures, the models answered single policy questions, such as a ban on nuclear power plants, correctly 72 to 89 percent of the time. The researchers used a simulation tool to compare the output of each scenario to each LLM’s answer. The results for combined policies are poor, as the models answer at a rate no better than random guessing when asked about pairs of policies. Accuracy drops when models must calculate how much a specific policy would change warming. The researchers grouped the open-weight models by the number of parameters in billions to track these performance trends.

Argumentative vulnerability

The 15 models tested in the experiment by Drobnjak, et al./UNESCO also show how easily they change their stance during debates. When a user identifies as a climate scientist, a policy adviser, or a climate skeptic to judge false claims like "CO2 is not the main driver of climate change," the initial identity does not affect the likelihood of a correct judgment. All 15 models relent to a significant degree when users provide sustained disagreement and counter-arguments over several rounds, which reduces their accuracy by 30 to 35 percent. I find the models’ susceptibility to pressure telling, especially since confidently restating a false claim as true works well when the user acts as a "climate scientist" but fails when the user claims to be a "climate skeptic." I conclude that these models fail to maintain factual integrity when faced with persistent user disagreement. Researchers tried more than just responding "nuh-uh" or "no it isn’t" to see if they could force a change in the model’s logic through various personas or persistent disagreement.