A source link is not enough to make an AI decision.

Why a credible source can still support a weak decision

Evidence has at least two dimensions. Origin asks where the information came from. Fit asks whether the evidence matches the claim and the decision.

Suppose a vendor reports that its assistant reduces response time by 40 percent in a demonstration. The source may be authentic and the number may be reported accurately. A decision to deploy the assistant across every customer conversation can still be unjustified.

The team still needs to know who performed the task, which questions were included, how response time was defined, what comparison was used, whether answer quality remained stable, and whether the conditions resembled daily operations. The source answers where the claim began. Scope determines how far the claim may travel.

The five levels of evidence weight

Level one is announcement. It can establish that an organization introduced, changed, or plans to offer something. The proportionate decision is watch. It does not prove value, savings, safety, or fit for your workflow.

Level two is documentation. It can establish intended behavior, controls, prerequisites, limits, and configuration choices. The proportionate decision is design. It does not prove performance with your people, data, policies, and failure patterns.

Level three is controlled test. It can establish what happened with a defined task set, method, comparison, and acceptance rule. The proportionate decision is pilot. It does not prove stable operation across every user and exception.

Level four is production record. It can establish how the workflow behaved during real use through volumes, errors, overrides, latency, escalation, source use, and policy exceptions. The proportionate decision is operate within the approved boundary. It does not prove that the customer or business benefited.

Level five is measured business outcome. It can establish whether the workflow improved the result behind the investment, such as faster resolution with stable quality, fewer missed appointments, lower rework, better conversion, or a lower error rate. The proportionate decision is expand, while the comparison, period, sample, and limitations remain visible.

What current assurance guidance adds

The NIST AI Risk Management Framework Core organizes risk work around Govern, Map, Measure, and Manage. Its Measure guidance covers documented methods, test sets, metrics, performance in conditions similar to deployment, production monitoring, and documented limits on generalization.

The NIST AI Metrology and Evaluation center provides access to metrics, methods, and tools organized by lifecycle and context. It also warns that inclusion does not establish NIST endorsement, validation, or suitability for a particular purpose. A credible collection is not a substitute for judging fit.

The United States Government Accountability Office framework connects accountability to governance, data, performance, and monitoring. United Kingdom guidance describes assurance as measuring, evaluating, and communicating trustworthiness through techniques selected for the context. Together, these sources support matching the method to the claim and consequence. They do not prescribe the five levels used here.

A real example of scope failure

The United States Federal Trade Commission final order concerning Workado addressed a claim that an AI detection product was 98 percent accurate. According to the agency, the cited testing concerned academic content, while independent testing found far lower performance on general purpose content.

The final order requires competent and reliable evidence for future accuracy claims and requires the company to retain that evidence when the claim is made.

The lesson is not that every vendor claim is false. The lesson is that evidence for one context should not be stretched across another context without proof.

A fictional service business example

Imagine a service company evaluating an AI assistant that drafts customer replies. The vendor states that the assistant cuts response time by 40 percent. At announcement level, the claim earns attention. Documentation can support a test design, but it cannot support a savings promise.

The company tests 120 fictional and historical questions with private details removed. It compares the current workflow with the proposed workflow and measures response time, factual errors, unsafe commitments, escalation, and reviewer effort. The assistant meets the acceptance rule on simple questions and fails on refund exceptions. The evidence supports a pilot limited to simple questions with mandatory review.

The production record then shows how often staff edit, reject, or escalate drafts during real work. Only after the company compares customer resolution time, repeat contact, quality review, and total handling effort against the prior baseline does it consider expansion.

Build an evidence receipt before the decision

Write the exact claim the decision depends on. Identify whether the support is an announcement, documentation, controlled test, production record, or measured outcome. Link to the primary source when available.

Record the people, task, data, conditions, comparison, metric, and period. State what the evidence does not prove. Then name the smallest proportionate action: watch, design, pilot, operate, or expand.

Finally, name the decision owner and the date when the evidence must be checked again. This receipt turns a source list into an operating control.

Where AI can help and where it should stop

AI can collect candidate sources, group evidence by claim, compare terminology, identify missing context, and draft an evidence receipt.

It should not silently decide that a source supports a broader claim than the source actually makes. A person should own the exact business claim, evidence relevance, acceptance rule, permitted decision, residual risk, and next review date.

This division keeps AI useful without turning citation into automatic authorization.

Opportunities, risks, and limits

Use the ladder in vendor evaluation by asking for the evidence class, method, context, and limitations behind important performance claims. Use it in internal proposals by showing whether each projected benefit comes from a source, calculation, test, or measured result.

Connect every evidence level to an approval boundary so a successful demonstration does not become an uncontrolled rollout. Refresh the evidence when the model, process, customer mix, policy, or operating conditions change.

The ladder can still create false confidence if teams reward the label but ignore the method. A controlled test can be badly designed. A production record can omit failures. A measured outcome can reflect another change. Sensitive decisions may require legal, security, privacy, technical, or domain review beyond this workflow.

Act now or wait

Act now when the next decision is small, reversible, measured, and supported by evidence that matches the intended context.

Wait when the evidence comes only from an announcement, the success metric is undefined, the operating conditions differ, the source cannot be reproduced, or the decision would create a difficult customer or compliance consequence.

The purpose of waiting is not caution for its own sake. It is to name the next proof required.

A practical 30, 60, and 90 day plan

Days 1 to 30: choose one AI claim behind a current proposal. Complete the evidence receipt. Remove any claim that cannot state its source, context, limitation, and permitted decision.

Days 31 to 60: run one controlled test with a baseline, representative task set, acceptance rule, failure categories, and named owner. Keep the rollout boundary explicit.

Days 61 to 90: compare the production record with the intended business outcome. Continue, revise, expand, or stop based on the evidence actually collected.