Dev48
Language
  • About
  • Services
  • Industries
  • Technologies
  • Articles
  • Contacts
Book a call
    Home/Articles/A checklist for trusting ai generated carbon footprints
Dev48
info@dev48.org+375 29 320-40-96WhatsApp@isky_by

© 2026 · All rights reserved.

A checklist for trusting AI-generated carbon footprints

Source: Watershed•November 18, 2026 at 10:16 PM

A checklist for trusting AI-generated carbon footprints

AI can now map tens of thousands of purchased materials to emissions factors in an afternoon. The open question is whether you can trust a particular result. New research from Watershed Science tackles it.

AI can now map tens of thousands of purchased materials to emissions factors in an afternoon, or build a cradle-to-gate footprint for a product that has never had one. The open question, for companies and researchers alike, is is how to decide whether you can trust a particular result. Today, we’re releasing a framework to give sustainability teams greater certainty in their measurements.

The question of how to test the trustworthiness of AI is harder in carbon accounting than in many fields, because there's rarely a right answer to check against. Six expert teams given the same cotton data landed 44% apart. Environmental product declarations (EPDs) for identical products vary by more than 50% under equally compliant assumptions. An AI system that reports "95% accuracy" is making a claim about agreement with somebody's judgment, and the useful questions are whose judgment, on which inputs, and what happens to the inaccurate 5%.

A paper we've just published open access in the Journal of Industrial Ecology proposes a set of evaluation criteria that answer those questions. We developed them with more than 60 sustainability, procurement, and LCA practitioners across eight enterprises and seven sectors, refined them with two verification firms, and then applied them to our own systems in Watershed Product Footprints. We hope the criteria are useful well beyond our own paper. This post walks through the criteria, how teams building AI footprinting systems can design around them, and how buyers and verifiers can use them to evaluate one. We look closely at two of them, indicator validation and repeatability, with results from our recent work including how we've applied them to Watershed's own AI product carbon footprinting systems.

Start with three questions

The criteria organize around three operational requirements. Each maps to a question a reviewer should ask.

  • How well does it perform against expert judgment or documented reference values? Benchmarking: Outputs should match expert-labeled acceptable mappings, reviewer-accepted modeling choices, or predefined tolerances, and the performance should be reported by category and input quality rather than as one blended figure.
  • Can an expert see how an output was generated? Transparent logic: The reasoning behind a mapping or a modeling decision has to be accessible and has to follow accepted carbon accounting principles. For a system built on a large language model (LLM), where you can't replay every internal step, that means what a user can see on each output: a structured summary of the reasoning, the calculation chain behind the number, which checks ran deterministically and which didn't, and citations for the sources that support it.
  • Which outputs need a human review? Uncertainty characterization and reviewability: The system shouldn't present guesses as facts. It should provide uncertainty indicators that have been validated, and documentation that lets a reviewer intervene on the material and high-risk items.

The first two have close analogues in software and machine learning validation: held-out test-set evaluation on one side, documented logic and audit trails on the other. The third is the one AI footprinting tools most often leave out, so it gets the most attention in the sections that follow.

The criteria for two use cases

We quickly found that a single checklist for all AI carbon footprinting use cases would be too strict for simple applications and too loose for complex ones, so the criteria are split by use case.

  • Case 1 is automated mapping: linking procurement text like "304 SS sheet" to a life cycle inventory dataset like "steel production, chromium steel 18/8, hot rolled."
  • Case 2 is automated modeling: generating a bill of materials and process flows for a product with no direct database match.

For automated mapping (Case 1), the criteria ask an AI system and its provider to demonstrate the following. The numbering follows the paper so you can cite it.

Criterion

What to look for

1.1 Benchmark performance

Accuracy against a validation set with expert-labeled acceptable mappings, with disclosure of the labeling process.

1.2 Consistency across material types and input formats

Performance reported by industry category and input complexity, with caveats where it isn't stable.

1.3 Appropriate granularity and proxy approach

Proxy selection logic that follows LCA practice when there's no exact match.

1.4 Repeatability

Unambiguous inputs get the same mapping across runs; inputs that don't are flagged as uncertain.

1.5 Match quality indication

A visible match-quality or uncertainty indicator that highlights poor matches for data improvement, human review, or escalation to Case 2.

1.5.1 Indicator validation

If that indicator is used to reduce or prioritize review, its error-detection performance is benchmarked against independent labels and disclosed.

1.6 to 1.10 Documentation

Mapping methodology, data sources, a change history for each mapping, decision logic, and an output-level review trail.

1.11 Continuous improvement

A mechanism for feeding corrections back into the system.

1.12 Position in the GHG Protocol hierarchy

A statement of where the method sits in the scope 3.1 data quality hierarchy.

PCF modeling systems carry the same documentation expectations plus their own benchmarking and uncertainty criteria.

Criterion

What to look for

2.1 Reasonable precision and component-level benchmarking

End-to-end checks against reference values such as EPDs, plus component-level checks on mass balances, input rates, and proxy choices, since totals alone hide offsetting errors.

2.2 Uncertainty indication

Confidence levels, ranges, or sensitivities for the datapoints and assumptions that drive the result.

2.2.1 Indicator validation

As in 1.5.1: any check used to waive or prioritize review is validated and its performance disclosed.

2.3 to 2.6 Documentation

Key data sources cited for the specific claim they support, the AI methodology, the calculation approach, and a statement of which uses the output is fit for and which it isn't.

2.7 Major assumptions

The assumptions that move the number, recorded.

2.8 Hotspot identification

Clear indication of the major emissions sources and improvement opportunities.

None of the criteria sets a pass mark, because whether 90% mapping accuracy is enough depends on whether the output feeds a screening-level estimate across thousands of low-materiality items or a reported figure held to a 5% materiality threshold. The criteria ask systems to disclose the evidence so that you, or your verifier, can make the fitness-for-purpose call.

Don't let the AI grade its own homework

One criterion we've found particularly important, as both builders and users of these systems, is indicator validation (1.5.1 and 2.2.1). It's also easy to miss. Most AI footprinting tools report some kind of confidence or quality score, and it's natural to take that score at face value until you test it.

Many customers, reasonably, want to use that score to decide what gets a human look. But in most systems the score comes from the same model family that produced the answer, often with the same prompt or within the same loop that made the decision. If you trust the score, you're trusting the AI to tell you when it's wrong, which is the circular trust problem.

The fix is to treat the indicator as a system output in its own right and benchmark it like one. Take a set of mappings with independent expert labels, run the indicator over them, and report the numbers a reviewer needs:

  • the share of erroneous outputs the indicator marks as acceptable, which is its false-negative rate if you auto-accept on it;
  • the error rate among the outputs it accepts, with a confidence interval;
  • the share of all errors it catches at the review budget you'd actually use.

Those numbers set the review policy. Until an indicator has been validated this way, review effort should be allocated by materiality and random sampling regardless of what the indicator says. Even once it has, the highest-materiality items should be reviewed no matter what the score reads.

Our own match-quality flag for the material auto-mapper illustrates both sides. Scored by an independent judge model that never sees the mapper's reasoning, it passed 1 of 97 errors in the validation set into the high-quality tier. Give the mapper's own model the judging job and that rises to 11 of 97. The component-level checks in our Advanced AI model, by contrast, haven't yet cleared this bar: the only reference values available are EPD totals, and against those the checks don't separate good graphs from bad ones. So we don't use them to waive review, and the paper says so.

Read repeatability as a diagnostic

Run-to-run variability is usually treated as a flaw in generative AI. The criteria treat it as useful information about the input. Run the mapper ten times on a procurement line that says only "Compartment boxes & lids" with the description "packaging", and it lands on injection-moulded polypropylene five times, corrugated board three times, and thermoformed polypropylene sheet twice. The system is telling you that the input doesn't support a single defensible answer. That is the item a human should look at, or a supplier should be asked to clarify. Run it ten times on "market for polypropylene, granulate", an exact database name, and all ten runs agree.

Identical runs certify stability. They say nothing about correctness, since a system can be perfectly repeatable and consistently wrong, so repeatability sits alongside benchmarking in the criteria rather than replacing it.

Putting this into practice doesn't mean running every item ten times. A purchasing dataset usually contains many similar or identical materials, and when those come back with different mappings, that is the same signal: the descriptions don't support a single defensible answer. Consider adding more descriptive details to those items.

What failing looks like

A framework demonstrated only on a system that passes isn't much use, so the paper also works through failures. The clearest come from PCFBench, the open benchmark we released for product carbon footprinting. General-purpose frontier models asked to build a footprint from components fall short on the benchmarking criteria, and their self-reported confidence couldn't support a review policy. The same red-flag pattern shows up across the criteria. Benchmark results come as a single number with no category breakdown, repeatability was never measured, and the confidence score has never been tested against anything.

Where our systems stand

We applied the criteria to the two systems in Watershed's product footprint tools: the material auto-mapper, which selects emissions factors for purchased materials, and the Advanced AI model, which builds cradle-to-gate footprints from a product description. The paper and its supplementary material give the criterion-by-criterion assessment.

If you've read our earlier post on how we benchmark the auto-mapper, the headline findings will be familiar: performance that holds up on well-specified inputs and drops on vague ones, which is itself a result the criteria require you to disclose. The specific figures move with every model and library update, and we expect them to. What we'd rather you take from the paper is the evidence structure around them, because that's what lets you evaluate the quality and defensibility of our numbers, or anyone else's.

Use the criteria

The paper is open access, with worked examples and the full validation evidence in the supplementary material. The criteria make a reasonable list of pre-requisites to ask for when you're evaluating an AI carbon footprinting system, from us or anyone else, and a reasonable list to publish when you're building one. Standards bodies working on product footprinting are welcome to adopt or improve them, and we'd like to help. Download a summary list of criteria here.

For more research on this topic, see our sustainability AI research brief, which collects PCFBench, the ATLAS spend-classification benchmark, and other research papers.

← All articles

What this article says

More in Data & Analytics

All →
Kilo Desktop for Data Science and Research Is Now Generally Available
Anaconda

Kilo Desktop for Data Science and Research Is Now Generally Available

How we fine-tuned a faster model for Patterns on Baseten
Datadog

How we fine-tuned a faster model for Patterns on Baseten

Introducing the Launch plan
Datalab

Introducing the Launch plan

Data Strategy vs AI Strategy: Are They the Same Thing?
Starburst

Data Strategy vs AI Strategy: Are They the Same Thing?

How Are Companies Handling AI Governance in Practice?
Starburst

How Are Companies Handling AI Governance in Practice?

Why AI Guardrails Matter
Starburst

Why AI Guardrails Matter

More from Watershed

Research brief: AI emissions
Watershed

Research brief: AI emissions

Research brief: Sustainability AI
Watershed

Research brief: Sustainability AI

Meet the second cohort of Watershed AI Fellows
Watershed

Meet the second cohort of Watershed AI Fellows

A catalog of starting use cases for AI in sustainability, plus two advanced workflows
Watershed

A catalog of starting use cases for AI in sustainability, plus two advanced workflows

Verdantix and Forrester named Watershed a leader six times in the last four years
Watershed

Verdantix and Forrester named Watershed a leader six times in the last four years

Forrester names Watershed a leader in sustainability management software
Watershed

Forrester names Watershed a leader in sustainability management software