Skip to content
← AI in practice / October–November 2026

AI product outcomes

How to Tell Whether an AI Feature Creates Business Value

Connect technical performance to customer outcomes and operating cost before deciding what deserves more investment.

Outcome Driven · Published · 5 min read

Illustrative arithmetic, not measured results: $200 ÷ 70 = $2.86; $130 ÷ 85 = $1.53 per accepted result. Total cost includes review and rework.
Illustrative arithmetic, not measured results: $200 ÷ 70 = $2.86; $130 ÷ 85 = $1.53 per accepted result. Total cost includes review and rework.
In this guide
  1. How do you measure useful AI work?
  2. Define the result before choosing the metric
  3. Connect the evidence across the product
  4. Count the cost of unsuccessful attempts
  5. Work through an illustrative comparison
  6. Build a comparison you can interpret
  7. Keep diagnostic measures separate from investment measures
  8. End the review with a decision
  9. Put this into practice
  10. Sources and further reading

How do you measure useful AI work?

Define a customer task and its acceptance criteria, compare against a baseline, and measure the complete cost of accepted outcomes—including review and rework. Use the results to decide whether to expand, change or pause the feature.

By Ashley Kays

An AI feature can produce better answers while the business sees little improvement. Customers may not discover it. Employees may spend too long checking its work. The system may handle easy cases well while making difficult cases more expensive.

For a B2B onboarding team, the useful test is whether a new customer reaches a usable first result with less effort. Start there before comparing model costs or output speed.

To decide whether an AI initiative deserves further investment, a team needs evidence that connects technical behavior to useful outcomes and the resources required to achieve them.

Spotify’s discussion of AI product management across three layers offers a helpful starting point. Research, shared capabilities, and user experiences can each create value, but that value becomes visible in different ways. My recommendation is to make those connections explicit in the measurement plan.

Define the result before choosing the metric

Start with the user’s task and the existing alternative. For a support assistant, a useful result might be a correctly resolved account-access issue. For an internal research assistant, it might be a decision brief whose important claims can be verified. For an onboarding assistant, it might be a customer completing a first useful project.

These definitions affect what counts as success. A support conversation that ends quickly is not necessarily resolved. A research summary that reads well may still require extensive checking.

Write down the acceptance conditions before reviewing results. Include how you will verify completion, the time window for follow-up, and which failures prevent a task from counting as successful.

Connect the evidence across the product

Use a small set of measures that explain different parts of the result.

QuestionExample evidenceWhat it helps you decide
Can the system do the taskPerformance on representative casesWhether the capability is ready for a limited pilot
Can people use it successfullyCompletion, correction effort, and repeat useWhether the workflow needs improvement
Does the result improveResolution quality or time to an accepted outputWhether the product helps the intended user
Is the improvement worth its costTotal operating cost and cost per accepted outcomeWhether to expand, redesign, or narrow the scope

Treat the connections as hypotheses until they are tested. Higher answer quality may reduce rework, but the measurement plan should establish whether that happens in the actual workflow.

Count the cost of unsuccessful attempts

Cost per generated answer leaves out much of the work an AI feature creates. Include the cost of review, corrections, retries, handoffs, and unresolved cases, as well as model and infrastructure costs.

For a defined group of tasks, one useful measure is:

Cost per accepted outcome = total delivery cost for all tasks in the group ÷ number of tasks meeting the acceptance conditions.

Keep the failed tasks in the cost numerator. Removing them would make a system that handles only easy cases look more economical than it is. If no outcomes qualify, report that directly; the ratio is undefined.

Separate initial implementation costs from ongoing operating costs, and include both when making an investment decision. Do not count time savings as cash savings unless the organization actually reduces spending or uses the released capacity in a measurable way.

Work through an illustrative comparison

Consider a fictional comparison of two support workflows. Each receives 100 eligible requests of similar difficulty, with the same definition of an accepted resolution and the same follow-up window. The numbers below are invented to demonstrate the calculation.

MeasureWorkflow AWorkflow B
Total requests100100
Accepted resolutions7085
Model and infrastructure cost$20$45
Review and handling cost for all requests$180$85
Total delivery cost$200$130
Cost per accepted resolution$2.86$1.53

Workflow B has a higher technical operating cost, but the complete workflow is less expensive per accepted resolution in this example. It also resolves a larger share of requests.

The practical lesson is to examine the complete cost of producing the desired result. This table does not establish that a more expensive model will improve a real product. A real test must check task mix, quality, staffing assumptions, and variability. Any additional adoption or implementation costs also belong in the investment decision.

Build a comparison you can interpret

Define who is eligible for the feature and how tasks enter the test. Where feasible, compare groups through random assignment. If that is not feasible, document relevant differences and describe the result as observational.

For example, employees who volunteer for an AI pilot may have stronger tool skills than those who continue with the existing process. A faster result could reflect that difference. Changes in request difficulty, seasonality, or staffing can also affect a before-and-after comparison.

Report the number of cases and the observation period alongside the result. Inspect important subgroups, especially cases with greater consequences or more complex requirements. An overall average can conceal a regression for the users who most need help.

Keep diagnostic measures separate from investment measures

A model evaluation helps explain behavior under defined conditions. A production measure helps establish what happened in the deployed workflow. Both are useful, but they answer different questions.

Anthropic’s evaluation guidance discusses automated evaluations alongside production monitoring, experiments, and user research. For an investment review, combine those perspectives rather than treating a single score as sufficient evidence.

If quality improves in testing but completion remains flat, investigate discovery, usability, integration, and whether the feature addresses a task users care about. If completion rises but rework increases, inspect the acceptance criteria and downstream consequences.

End the review with a decision

An effective review should identify what the team now knows, what remains uncertain, and what happens next.

Before the pilot, agree on the evidence that would support expansion, the failures that would trigger a pause, and the questions that would justify another focused test. Set thresholds appropriate to the task’s consequences and the strength of available evidence. A fixed percentage borrowed from another product is unlikely to be useful.

Then report the outcome in ordinary language: more people completed the task, the cost changed by this amount, these failures remain, and this is the next investment we recommend.

That explanation gives leaders a basis for allocating resources and gives the team a clear reason for its next piece of work.

Put this into practice

Use the expertise pilot playbook to document the outcome, baseline and next decision. Pair it with From signup to first value to locate the customer moment you want to improve. For evaluation ownership, read Who Owns Quality in an AI Product.

The measurement framework and financial example are original analytical tools. The example contains no observed client or Spotify results.

Sources and further reading

Sources checked November 16, 2026. Review the linked publisher information for current details.