AI can now map tens of thousands of purchased materials to emissions factors in an afternoon, or build a cradle-to-gate footprint for a product that has never had one. The open question, for companies and researchers alike, is is how to decide whether you can trust a particular result. Today, we’re releasing a framework to give sustainability teams greater certainty in their measurements.
The question of how to test the trustworthiness of AI is harder in carbon accounting than in many fields, because there's rarely a right answer to check against. Six expert teams given the same cotton data landed 44% apart. Environmental product declarations (EPDs) for identical products vary by more than 50% under equally compliant assumptions. An AI system that reports "95% accuracy" is making a claim about agreement with somebody's judgment, and the useful questions are whose judgment, on which inputs, and what happens to the inaccurate 5%.
A paper we've just published open access in the Journal of Industrial Ecology proposes a set of evaluation criteria that answer those questions. We developed them with more than 60 sustainability, procurement, and LCA practitioners across eight enterprises and seven sectors, refined them with two verification firms, and then applied them to our own systems in Watershed Product Footprints. We hope the criteria are useful well beyond our own paper. This post walks through the criteria, how teams building AI footprinting systems can design around them, and how buyers and verifiers can use them to evaluate one. We look closely at two of them, indicator validation and repeatability, with results from our recent work including how we've applied them to Watershed's own AI product carbon footprinting systems.
Start with three questions
The criteria organize around three operational requirements. Each maps to a question a reviewer should ask.
- How well does it perform against expert judgment or documented reference values? Benchmarking: Outputs should match expert-labeled acceptable mappings, reviewer-accepted modeling choices, or predefined tolerances, and the performance should be reported by category and input quality rather than as one blended figure.
- Can an expert see how an output was generated? Transparent logic: The reasoning behind a mapping or a modeling decision has to be accessible and has to follow accepted carbon accounting principles. For a system built on a large language model (LLM), where you can't replay every internal step, that means what a user can see on each output: a structured summary of the reasoning, the calculation chain behind the number, which checks ran deterministically and which didn't, and citations for the sources that support it.
- Which outputs need a human review? Uncertainty characterization and reviewability: The system shouldn't present guesses as facts. It should provide uncertainty indicators that have been validated, and documentation that lets a reviewer intervene on the material and high-risk items.
The first two have close analogues in software and machine learning validation: held-out test-set evaluation on one side, documented logic and audit trails on the other. The third is the one AI footprinting tools most often leave out, so it gets the most attention in the sections that follow.
The criteria for two use cases
We quickly found that a single checklist for all AI carbon footprinting use cases would be too strict for simple applications and too loose for complex ones, so the criteria are split by use case.
- Case 1 is automated mapping: linking procurement text like "304 SS sheet" to a life cycle inventory dataset like "steel production, chromium steel 18/8, hot rolled."
- Case 2 is automated modeling: generating a bill of materials and process flows for a product with no direct database match.
For automated mapping (Case 1), the criteria ask an AI system and its provider to demonstrate the following. The numbering follows the paper so you can cite it.
Criterion
What to look for
1.1 Benchmark performance
Accuracy against a validation set with expert-labeled acceptable mappings, with disclosure of the labeling process.
1.2 Consistency across material types and input formats
Performance reported by industry category and input complexity, with caveats where it isn't stable.
1.3 Appropriate granularity and proxy approach
Proxy selection logic that follows LCA practice when there's no exact match.
1.4 Repeatability
Unambiguous inputs get the same mapping across runs; inputs that don't are flagged as uncertain.
1.5 Match quality indication
A visible match-quality or uncertainty indicator that highlights poor matches for data improvement, human review, or escalation to Case 2.
1.5.1 Indicator validation
If that indicator is used to reduce or prioritize review, its error-detection performance is benchmarked against independent labels and disclosed.
1.6 to 1.10 Documentation
Mapping methodology, data sources, a change history for each mapping, decision logic, and an output-level review trail.
1.11 Continuous improvement
A mechanism for feeding corrections back into the system.
1.12 Position in the GHG Protocol hierarchy
A statement of where the method sits in the scope 3.1 data quality hierarchy.
PCF modeling systems carry the same documentation expectations plus their own benchmarking and uncertainty criteria.
Criterion
What to look for
2.1 Reasonable precision and component-level benchmarking
End-to-end checks against reference values such as EPDs, plus component-level checks on mass balances, input rates, and proxy choices, since totals alone hide offsetting errors.
2.2 Uncertainty indication
Confidence levels, ranges, or sensitivities for the datapoints and assumptions that drive the result.
2.2.1 Indicator validation
As in 1.5.1: any check used to waive or prioritize review is validated and its performance disclosed.
2.3 to 2.6 Documentation
Key data sources cited for the specific claim they support, the AI methodology, the calculation approach, and a statement of which uses the output is fit for and which it isn't.
2.7 Major assumptions
The assumptions that move the number, recorded.
2.8 Hotspot identification
Clear indication of the major emissions sources and improvement opportunities.
None of the criteria sets a pass mark, because whether 90% mapping accuracy is enough depends on whether the output feeds a screening-level estimate across thousands of low-materiality items or a reported figure held to a 5% materiality threshold. The criteria ask systems to disclose the evidence so that you, or your verifier, can make the fitness-for-purpose call.
Don't let the AI grade its own homework
One criterion we've found particularly important, as both builders and users of these systems, is indicator validation (1.5.1 and 2.2.1). It's also easy to miss. Most AI footprinting tools report some kind of confidence or quality score, and it's natural to take that score at face value until you test it.
Many customers, reasonably, want to use that score to decide what gets a human look. But in most systems the score comes from the same model family that produced the answer, often with the same prompt or within the same loop that made the decision. If you trust the score, you're trusting the AI to tell you when it's wrong, which is the circular trust problem.
The fix is to treat the indicator as a system output in its own right and benchmark it like one. Take a set of mappings with independent expert labels, run the indicator over them, and report the numbers a reviewer needs:
- the share of erroneous outputs the indicator marks as acceptable, which is its false-negative rate if you auto-accept on it;
- the error rate among the outputs it accepts, with a confidence interval;
- the share of all errors it catches at the review budget you'd actually use.
Those numbers set the review policy. Until an indicator has been validated this way, review effort should be allocated by materiality and random sampling regardless of what the indicator says. Even once it has, the highest-materiality items should be reviewed no matter what the score reads.
Our own match-quality flag for the material auto-mapper illustrates both sides. Scored by an independent judge model that never sees the mapper's reasoning, it passed 1 of 97 errors in the validation set into the high-quality tier. Give the mapper's own model the judging job and that rises to 11 of 97. The component-level checks in our Advanced AI model, by contrast, haven't yet cleared this bar: the only reference values available are EPD totals, and against those the checks don't separate good graphs from bad ones. So we don't use them to waive review, and the paper says so.
Read repeatability as a diagnostic
Run-to-run variability is usually treated as a flaw in generative AI. The criteria treat it as useful information about the input. Run the mapper ten times on a procurement line that says only "Compartment boxes & lids" with the description "packaging", and it lands on injection-moulded polypropylene five times, corrugated board three times, and thermoformed polypropylene sheet twice. The system is telling you that the input doesn't support a single defensible answer. That is the item a human should look at, or a supplier should be asked to clarify. Run it ten times on "market for polypropylene, granulate", an exact database name, and all ten runs agree.
Identical runs certify stability. They say nothing about correctness, since a system can be perfectly repeatable and consistently wrong, so repeatability sits alongside benchmarking in the criteria rather than replacing it.
Putting this into practice doesn't mean running every item ten times. A purchasing dataset usually contains many similar or identical materials, and when those come back with different mappings, that is the same signal: the descriptions don't support a single defensible answer. Consider adding more descriptive details to those items.
What failing looks like
A framework demonstrated only on a system that passes isn't much use, so the paper also works through failures. The clearest come from PCFBench, the open benchmark we released for product carbon footprinting. General-purpose frontier models asked to build a footprint from components fall short on the benchmarking criteria, and their self-reported confidence couldn't support a review policy. The same red-flag pattern shows up across the criteria. Benchmark results come as a single number with no category breakdown, repeatability was never measured, and the confidence score has never been tested against anything.
Where our systems stand
We applied the criteria to the two systems in Watershed's product footprint tools: the material auto-mapper, which selects emissions factors for purchased materials, and the Advanced AI model, which builds cradle-to-gate footprints from a product description. The paper and its supplementary material give the criterion-by-criterion assessment.
If you've read our earlier post on how we benchmark the auto-mapper, the headline findings will be familiar: performance that holds up on well-specified inputs and drops on vague ones, which is itself a result the criteria require you to disclose. The specific figures move with every model and library update, and we expect them to. What we'd rather you take from the paper is the evidence structure around them, because that's what lets you evaluate the quality and defensibility of our numbers, or anyone else's.
Use the criteria
The paper is open access, with worked examples and the full validation evidence in the supplementary material. The criteria make a reasonable list of pre-requisites to ask for when you're evaluating an AI carbon footprinting system, from us or anyone else, and a reasonable list to publish when you're building one. Standards bodies working on product footprinting are welcome to adopt or improve them, and we'd like to help. Download a summary list of criteria here.
For more research on this topic, see our sustainability AI research brief, which collects PCFBench, the ATLAS spend-classification benchmark, and other research papers.








