ANTALYA, Türkiye — The world's leading AI chatbots answer basic climate questions well, but they abandon established climate facts as soon as users push back, judge evidence poorly, and largely fail when a question combines two climate policies, according to a UNESCO and University College London (UCL) report, "Can We Trust AI on Climate?", presented on Monday, November 16, at the COP31 climate conference in Antalya.
The study, led by UCL researchers and published by UNESCO, examined the reliability of climate information from up to 33 leading AI large language models, tested five of them across 13 widely spoken languages from English to Urdu, and put 656 climate-policy questions to the systems. It was prepared for UNESCO's Global Initiative for Information Integrity on Climate Change, which UNESCO co-chairs with the UN and Brazil.
Right on the basics, weak on judgement
On basic climate-change facts, the best-performing AI systems answered accurately between 88% and 96% of the time, depending on the test. But when the models were asked to evaluate whether a specific piece of evidence supports or refutes a scientific claim, even the best managed only 71% to 77% accuracy.
The collapse was sharpest on complex reasoning. Asked to assess combinations of climate policies — for example a carbon tax paired with subsidies — accuracy plunged to 33.8%, no better than random guessing, The Hindu BusinessLine reported. Individual policy interventions, by contrast, were answered with up to 94.5% accuracy.
"Sycophancy": eight in ten climbdowns came at the first pushback
In a separate set of tests on 15 models, researchers challenged answers the models had initially got right, moving from simple disagreement to claimed expertise and fabricated evidence. Across all 15, accuracy dropped by between 31 and 35 percentage points under that pressure, and about 80% of the concessions — roughly eight in ten — happened at the very first sign of disagreement. Every model tested eventually gave up a correct answer and adopted the user's incorrect position, a behaviour the researchers call AI sycophancy.
"This sycophantic behaviour, where models become more agreeable even at the expense of established facts, is a serious concern for the integrity of climate information," said Leona Verdadero, a programme specialist at UNESCO and the report's lead author there, according to The Hindu BusinessLine. The UCL team was led by Professor Ivana Drobnjak of UCL Computer Science, a member of the UNESCO Chair in AI at UCL.
A language gap that maps onto the climate front line
Accuracy also depended on the language of the question. Across the five models tested in 13 languages, average accuracy reached 88.6% in English, with Spanish, French and Portuguese close behind, but fell to 83.6% in Hindi and Bengali, 81.4% in Urdu and 78.3% in Swahili, UN News reported. For readers in Punjab and across India — where Hindi is a main language for asking AI questions and climate impacts from heat to floods are immediate — the report's finding is direct: the same chatbot gives measurably weaker climate answers in Hindi than in English, before any pushback even begins.
The report landed at a summit where AI is being promoted as part of the climate solution: COP31's hosts launched the Antalya AI Pledge on green digital action the same week. UN climate chief Simon Stiell warned the summit that AI spreading disinformation and undermining science would damage public trust at the moment that trust is most needed, according to reporting on the launch.
What is solid
- The report "Can We Trust AI on Climate?" was presented by UNESCO and UCL at COP31 in Antalya on November 16
- The study examined up to 33 leading AI models, five of them across 13 languages, using 656 climate-policy questions
- In pushback tests on 15 models, accuracy fell 31–35 percentage points and about 80% of concessions came at the first disagreement
What is only claimed
- How far controlled test results predict everyday chatbot behaviour — the study measured model responses in designed experiments, and the AI companies behind the models had not publicly responded to the findings in the coverage available at publication
The same summit also launched a voluntary Antalya AI Pledge on greener digital action — the pledge story is separate from this report's findings. Read the COP31 Antalya AI Pledge story
What happens next: UNESCO says the report is meant to feed its information-integrity work with governments and platforms. Whether AI companies change how their models handle disagreement — the training methods that reward agreeable answers — is the open question the report leaves on the table.
