AI has the potential to accelerate the work of sustainability teams by automating time-consuming parts of emissions measurement and surfacing new paths to decarbonization. But for AI to advance sustainability, teams first need to be able to trust and rely on its outputs.
In the rush to deploy AI, some sustainability teams are adopting tools without a real method for evaluating their reliability. Our research develops open evaluation methods that help technology companies build reliable AI systems for sustainability, and help sustainability teams make informed decisions about where AI fits the complexity of their work.
Watershed research on sustainability AI
Benchmarks: Evaluating AI accuracy for sustainability tasks and workflows
PCFBench: A diagnostic benchmark for product carbon footprint estimation Preprint, 2026 An open benchmark that tests AI models on each step of building a product carbon footprint, from breaking a product down into its materials to matching those materials with emissions data. It includes 614 expert-labeled examples and results for eight frontier AI models. The dataset and evaluation code are freely available.
ATLAS: A spend classification benchmark for estimating scope 3 carbon emissions NeurIPS 2024, Tackling Climate Change with Machine Learning workshop (spotlight paper) Most companies that report supply chain emissions start from their spending data. ATLAS is a set of 10,000 labeled spend items for testing how accurately AI assigns each purchase to the right emissions category.
Evaluation criteria: How to build credible AI systems for sustainability
Evaluation criteria for AI-assisted product carbon footprinting systems: The cases of mapping and supply chain modeling Preprint, 2026; in review at Journal of Industrial Ecology Defines what a credible AI carbon footprinting system should be able to demonstrate, including benchmark performance, uncertainty indicators, and a record of its reasoning that a reviewer can inspect. The criteria were developed with corporate sustainability practitioners and two verification firms, then applied to two AI systems in use at Watershed.
Quality-aware automation for LCI database mapping Journal of Industrial Ecology, 2026 Matching thousands of purchased materials to emissions data takes experts minutes per item, and a wrong match can look as plausible as a right one. This paper pairs an AI that proposes each match with a second, independent AI that scores it, so reviewers can concentrate on the matches most likely to be wrong.
How are companies reducing emissions? An LLM-based approach to creating a carbon emissions reduction levers library at scale NeurIPS 2024, Tackling Climate Change with Machine Learning workshop (proposal) Information about how companies are cutting emissions is scattered across long sustainability reports. This proposal outlines an AI system that extracts reduction actions from public reports and organizes them by industry, region and emissions scope, so companies can see which actions are in use in their sector.
Build evals you can sustain Watershed, 2025 A practical framework for evaluating AI systems that complete work in multiple steps.
Key findings
Across our benchmarks and system tests, the importance of choosing the right evaluation framework was evident. Scoring only the final number hides errors that step-by-step evaluations expose, and the design choices that make a system reliable often only become visible when each step is measured. By testing the test as we've done in some of this research, we're building toward increasingly better evaluation systems for building increasingly precise AI.
Where AI breaks down in carbon accounting
- Accuracy drops when models show their work. In tests of eight frontier AI models, the strongest estimated a product's total emissions within 2× of its verified value for 77% of products. When the same models built the footprint step by step, that fell to 37–58%. (PCFBench, 2026)
- Many AI-generated bills of materials don't add up. In the same tests, 25–55% of the bills of materials the models produced failed mass conservation: the component weights didn't match the weight of the product. (PCFBench, 2026)
- Confidence was reliable on some tasks and not others. When models matched materials to emissions data, their stated confidence roughly tracked their accuracy. When they had to decide whether a material could be matched at all or needed to be broken down further, their most confident answers (about 95% stated confidence) were right about 63% of the time. (PCFBench, 2026)
- Vague inputs undermine accuracy. Watershed's AI mapping system matched expert-accepted choices for more than 90% of well-specified materials, but for 59.7% of vague descriptions such as "plastic part." (Evaluation Criteria, 2026)
Design choices that improved reliability in our tests
- An independent checker makes errors easier to find. In a system where one AI maps materials to emissions data and a second AI scores each match without seeing the first one's reasoning, the checker caught 67% of errors while reviewing 20% of items, compared with 37–40% for simpler flagging methods. Its scores allowed 30% of mappings to be accepted automatically with a 0.3% error rate. (Quality-aware automation, 2026)
- Domain expertise raises accuracy. Removing expert-written guidance on choosing proxy datasets cut the mapping system's accuracy by about 9 percentage points, with the rest of the system unchanged. (Quality-aware automation, 2026)
- Variation across repeat runs can flag vague inputs. Run repeatedly on the same input, the mapping system gave identical answers for exact matches and different answers for vague inputs (polyethylene one time, polypropylene the next, for "plastic part"). That variation can be used to route items to human review. (Evaluation Criteria, 2026)
- Checking each component shows how to prioritize reviews. Benchmarked against 269 third-party-verified environmental product declarations, AI-generated product footprints showed a 33% median error with no systematic bias. The paper notes that declarations for identical products can differ by more than 50% under equally compliant methods. Component-level checks found material and energy input rates were the weakest part of the models. (Evaluation Criteria, 2026)
PCFBench tests general-purpose AI models. The Evaluation Criteria and Quality-aware automation papers tested Watershed's own systems, and all authors are Watershed employees, as the papers disclose. Each study uses its own metrics and datasets, so figures from different studies shouldn't be compared directly.
What this means for sustainability teams
As you evaluate AI systems for sustainability work, it's important to know what questions to ask and what to look for in those systems.
Questions to ask before relying on any AI system for emissions work:
- Is the system evaluated at each step, or only on the final number?
- What happens when there's no good match in the emissions data? Does the system flag it or pick something anyway?
- Has its confidence been checked against expert review?
- Is quality checked by a separate process from the one that produced the answer?
- How does accuracy change when inputs are vague, and does the system tell you which inputs are vague?
- Is the system revalidated when the underlying AI model changes?
What to look for
- Intermediate steps you can inspect. For a product footprint, that means the bill of materials, the emissions data each material was matched to, and the energy and material input rates, each with a source.
- A quality or confidence indicator on every item, with weak matches clearly flagged for review.
- Results broken down by task and input quality. A credible provider can show how the system performs on well-specified versus vague inputs, and on each step of the process.
- Basic physical checks. In a product footprint, component weights should add up to the weight of the product.
- Documentation of data sources and versions, including which emissions database and which AI model produced each result, so results can be reproduced.
- A plain statement of limitations and the uses the system is and isn't suited for.
About the researchers
Shaena Ulissi, Data & Climate Scientist, Lead author on Watershed's evaluation criteria for AI-assisted carbon footprinting. Research: Evaluation Criteria for AI-Assisted Product Carbon Footprinting Systems (lead author); PCFBench; Quality-aware automation; emissions reduction levers library.
Andrew Dumit, AI Engineer. Builds AI systems that measure product carbon footprints and supply chain emissions. Research: Quality-aware automation, ATLAS, PCFBench, Evaluation Criteria; Build evals you can sustain.
Krishna Rao, AI Scientist. Lead author on PCFBench. Research: PCFBench (lead author); ATLAS; Quality-aware automation; Evaluation Criteria.
Dr. James Joyce, Sustainability Science Lead. Life cycle assessment specialist. Research: Evaluation Criteria; PCFBench; Quality-aware automation.
Steven Watson, Product Lead, Watershed Product Footprints. Research: PCFBench; Evaluation Criteria; Quality-aware automation.
Dr. Sangwon Suh, Tsinghua University, research advisor to Watershed. Research: Evaluation Criteria; ATLAS; emissions reduction levers library.
Stay in the loop
Email*
By submitting this form you agree to Watershed’s privacy policy, and consent to receive Watershed marketing communications.
