Evidence for AI workflows · An experiment guide
An agent can use fewer calls, finish sooner and still produce the wrong answer. Evaluate a connected workflow at the point where a person can accept its result.
A team adds an MCP connection to its research assistant. The demonstration is impressive: the assistant locates documents, extracts details and assembles a recommendation. Someone asks whether it is ready to use every week. A recording of the successful demo cannot answer that question. You need a defined task, a credible comparison and a scoring rule that survives inconvenient results.
This guide turns that question into a small, inspectable experiment. The accompanying Agent Workflow Evidence Kit includes a hypothesis sheet, paired test records, a scoring rubric and a decision memo. Its fictional dataset deliberately contains a missing time and a quality failure so you can practice reporting useful results without smoothing away the uncertainty.
1. Choose the decision before choosing the metric
Begin with a real product decision: should a team pilot an assistant that prepares an evidence brief from an approved document collection? A brief is acceptable only if it answers the request, links to relevant sources, represents those sources accurately and states what remains unknown.
Now define the unit of work. One trial is one request carried through to a reviewed brief, including corrections. It is not one tool call, one model response or one retrieved document. Keep those lower-level measures as diagnostic signals. They help explain a result but should not replace the result the user needs.
Choose a comparison that reflects the current workflow. In this example, a person searches the same approved collection and prepares the brief using the team’s usual process. The candidate is the same task with the MCP-assisted workflow. If you also change the collection, model, prompt and output format, record those differences. You are testing a bundle, not isolating the effect of MCP.
Write a decision rule before seeing the outcomes. For a small exploratory run, it may be enough to decide which failure to fix before a wider pilot. Avoid dressing a tiny convenience sample as proof of a general improvement. The scope of your conclusion should fit the work you actually observed.
2. Know what the connection changes
MCP gives an AI application a structured way to access capabilities exposed by a server. That server can retrieve records or run an existing business function. It does not, by itself, establish whether the assistant used the right evidence or whether the user’s task succeeded.
WebMCP makes named actions available in an open webpage. Its current Chrome implementation uses document.modelContext; support is experimental. When evaluating a browser experience, keep a normal UI path and record the browser and API availability. Otherwise, a compatibility difference can masquerade as a workflow result.
For this experiment, begin with one read action: retrieve approved evidence for a specified topic. Keep source eligibility rules explicit. Require returned records to include stable IDs, source links and the attributes needed for review. The evaluator should be able to inspect what the assistant actually received.
The MCP architecture documentation and Chrome’s WebMCP reference explain the technical layers. The measurement approach below is a proposed working method for your team, not a benchmark supplied by those projects.
3. Write a one-page experiment agreement
Describe the user, task, baseline, candidate and eligible cases. State the primary outcome in plain language: “A reviewer accepts the brief within the trial deadline, with all required evidence and no critical factual error.” Define what counts as an error before the reviewer sees the outputs.
Choose secondary measures that explain the primary result: total elapsed time, human review minutes, correction count and direct run cost when available. Start the clock at the same point in both conditions. End it after review and necessary corrections, not when the assistant prints its first answer.
Set the deadline and missing-data policy in advance. A run that is unresolved at the deadline is not an accepted result for the primary outcome. Its unknown completion time remains missing. Do not enter zero, and do not replace it with the average of successful runs.
Write down the planned exclusions. An accidental duplicate request may be excluded if the rule applies equally to both conditions. A difficult request should not disappear because the candidate handled it poorly. Maintain an exclusion log with the reason and original case ID.
4. Build a test set that can expose a weakness
Use representative work, not only requests that match your demonstration. Include a straightforward lookup, a question spanning two documents, conflicting versions, an absent answer, an inaccessible source, an ambiguous request and a stale record. The mixture should reflect the users and decisions you intend to support.
Pair the baseline and candidate on the same underlying cases. Where a person runs both, vary the order or use separate people to reduce the advantage of having just learned the answer. Record the order rather than assuming pairing removes every source of bias.
Freeze a source snapshot when practical. If the live collection changes, record the relevant version or retrieval time. A candidate that finds newer information may be useful, but that is a different comparison from two workflows operating on identical evidence.
Create an answer key for facts that can be checked and a rubric for judgments that cannot. Ask a second reviewer to score a subset independently. Disagreement is a signal that the definition of acceptance needs work; do not average away a critical factual dispute without resolving it.
5. Run the connection and capture the whole task
For an existing MCP integration, use the provider’s documented connection method in a compatible assistant. Confirm the intended source and available tools. First request a known record and verify it against the source. Then request something absent. These checks establish that the experiment is using the intended connection before you measure it.
If you need a custom server, write the retrieval contract first: accepted query fields, source eligibility, returned evidence, empty-result behavior and access checks. Implement the ordinary retrieval function, wrap it in a tool using an official SDK and test the function and client call separately. The AI Teachers implementation lab supplies a small runnable starting point.
During each trial, preserve the request, tool result, final artifact and review notes. Record model and tool versions, source snapshot, elapsed minutes and corrections. A screenshot of the final answer alone may hide an earlier failed attempt or a manual repair.
Keep a human intervention field. “The reviewer supplied the missing document” is an important part of the workflow’s cost. Count that work in the trial rather than attributing the finished result entirely to the agent.
6. Score evidence quality separately from speed
| Dimension | Accepted behavior | Failure to record |
|---|---|---|
| Coverage | Answers every required part or explicitly identifies an unavailable answer | Silently skips a required question |
| Grounding | Each material factual claim points to relevant evidence | Cites a source that does not support the claim |
| Currency | Uses the eligible version and states the relevant date | Treats a superseded document as current |
| Boundaries | Uses permitted sources and preserves required human decisions | Uses an inaccessible collection or performs an unrequested write |
| Reviewability | A reviewer can inspect the sources and understand uncertainty | Produces a confident answer with no inspectable evidence |
Use a clear pass condition rather than an attractive average score. A brief with excellent formatting and an invented claim should not pass because presentation points compensate for factual failure. Mark critical failures explicitly and retain the reviewer’s explanation.
Distinguish tool success from task acceptance. A retrieval call can return valid JSON while omitting the document that matters. Conversely, a first call can fail and the workflow can recover, with the additional time included. These distinctions make your diagnosis useful to the person improving the system.
7. Work through an imperfect result
The following eight paired cases are fictional teaching data. They describe reviewed evidence briefs, not measured results from a deployed product. All listed durations include human review and corrections. Case 8 in the candidate condition remained unresolved at the deadline, so its completion time is unknown.
| Case | Baseline minutes | Candidate minutes | Baseline accepted | Candidate accepted |
|---|---|---|---|---|
| 1 | 8 | 5 | Yes | Yes |
| 2 | 12 | 8 | Yes | Yes |
| 3 | 7 | 6 | Yes | Yes |
| 4 | 10 | 8 | No | Yes |
| 5 | 9 | 7 | Yes | No: stale evidence |
| 6 | 13 | 10 | Yes | Yes |
| 7 | 6 | 7 | No | Yes |
| 8 | 11 | Missing | Yes | No: unresolved |
The primary acceptance rate is 6 ÷ 8 = 75% for both conditions. The candidate rescues cases 4 and 7 but loses cases 5 and 8. Reporting only its successful examples would hide that exchange.
For the seven pairs with a known duration in both conditions, baseline time totals 65 minutes and candidate time totals 51. The reduction on that matched subset is (65 − 51) ÷ 65 × 100 = 21.5%, rounded to one decimal place. Label it “elapsed-time reduction on seven complete pairs.” It is not an estimate of total time saved across all eight tasks.
The baseline total for all eight is 76 minutes. The candidate total is unknown, so an all-case time comparison is unavailable. We also cannot calculate cost per accepted brief without complete cost records. The correct entry for that metric is “not measured,” not a guess based on fewer tool calls.
8. Turn the evidence into the next experiment
This fictional run supports a specific conclusion: the candidate was faster on the complete timing subset, while overall acceptance was unchanged and two candidate weaknesses need investigation. It does not establish statistical significance, dependable savings or readiness for unattended use.
A reasonable next decision is to fix source-version selection and diagnose the unresolved case, then run the frozen cases again plus new cases that probe those failure modes. Keep the original results. The next run is a new version of the experiment, not a replacement for the inconvenient first one.
Make the decision memo short enough to use: decision, evidence, limitations, changes, owner and next review date. Attach case IDs to each claim. If a stakeholder asks, “Why are we expanding this?”, the answer should point to accepted tasks and observed user value.
Use the Agent Workflow Evidence Kit
Fill the hypothesis sheet before connecting a new tool. Adapt the scoring rubric to one task. Replace the fictional records with your own cases while preserving null values for missing observations. After the run, use the decision memo to separate what you measured from what you still need to learn.
The kit is deliberately small enough for a working team to use in a single pilot. It will not decide your release for you. It will make the reasoning visible, keep the comparison honest and give the next builder a precise problem to solve.
Technical references
- Official MCP architecture
- Chrome: WebMCP imperative API
- AI Teachers: implementation lab and tested starter
Explore another application
- The Wealth Experiment: From one useful AI connection to a pocket product people can evaluate
- AI Side Hustle: Build your first MCP-assisted client service in a seven-day sprint
- Waymaker: From idea to connected MVP: build one MCP tool people can use
- MarketingCoach: Give your marketing agent the evidence before you give it the controls
- CodeTeachers: Build a resource-finder MCP server—and prove it works

