Use AI in product development without skipping the evidence

In this guide

1. The useful starting point
2. Write the problem without describing your solution
3. Use a four-part evidence ledger
4. A fictional example: a handoff assistant
5. Move through five small decisions
6. Use AI as a challenging product partner
7. Test the experience, not only the output
8. When the first test disappoints
9. Questions you might be asking
10. Sources and further reading

The useful starting point

Use AI to organize evidence, explore alternatives, and prepare prototypes. Keep generated personas and simulated feedback labeled as hypotheses, then test the important assumptions with real people and observable behavior before expanding the product.

AI can make an idea feel real remarkably quickly. A polished screen, a plausible user story, and a confident roadmap can arrive before anyone has confirmed that the problem matters. The speed is useful when it shortens the path to learning.

A strong product workflow uses that speed to expose uncertainty. It asks what must be true, creates the smallest useful test, and gathers evidence that could change the team's direction.

Write the problem without describing your solution

Describe a person, a situation, an obstacle, and its consequence. For example: “A project coordinator receives incomplete handoff notes and spends time asking follow-up questions before work can begin.” That statement leaves room for several solutions.

Now separate evidence from interpretation. A recorded example of missing information is evidence. “Everyone needs an AI handoff agent” is a proposed explanation and solution. Keeping those apart helps the team avoid turning its preferred technology into the problem definition.

Ask AI to identify missing information in the problem statement. It may help you notice that you do not know the frequency, severity, current workaround, or person responsible for fixing the issue.

Use a four-part evidence ledger

All four can be useful, but they carry different weight. An invented persona can help you explore questions. It is not a research participant. A synthetic interview can reveal gaps in your script. It cannot establish how real customers feel.

A fictional example: a handoff assistant

Imagine a small service team exploring an AI tool that turns sales notes into delivery briefs. The first prototype produces a beautiful document. That alone does not establish value.

The team collects several approved, redacted examples and asks delivery colleagues what information they need to begin work. They discover that the biggest problem is not formatting. It is uncertainty about what was promised and which details still require customer confirmation.

The prototype changes. It now separates confirmed scope, unresolved questions, and proposed next steps. A human approves the brief before it reaches delivery. In a limited test, the team records corrections, missing commitments, and time spent reviewing, rather than simply asking whether people like the design.

Those observations can support a next decision. If reviewers repeatedly rewrite the same section, the team can change the input template or the product behavior. If the source notes are consistently incomplete, a better sales capture process may be more useful than a more elaborate model.

Move through five small decisions

1. Choose the risky assumption. Identify what would make the idea unhelpful if false.
2. Select a test format. A conversation, paper prototype, manual service, or clickable flow may answer the question without a full application.
3. Define observable success. State what the person should accomplish and what errors would matter.
4. Run the test with appropriate participants. Give them a realistic task and observe without coaching them toward your preferred answer.
5. Decide what changes. Continue, revise, investigate further, or stop based on the evidence.

Set the decision criteria before seeing the results. Otherwise, it is easy to redefine success around whatever happened. Keep the first test small enough that changing direction remains practical.

Use AI as a challenging product partner

Review this problem statement and evidence ledger. Identify the most important untested assumption and three alternative solutions, including one that uses no AI. Design a small test that could show our preferred solution is wrong. Keep generated examples separate from real evidence. Suggest observable outcomes, likely failure modes, and questions for actual participants. Do not invent research findings.

Ask for the strongest case against the idea, then inspect whether that case relies on real constraints or generic pessimism. A useful sounding board improves the questions; it does not simply agree or disagree theatrically.

Test the experience, not only the output

For an AI feature, evaluate how people provide context, inspect results, correct mistakes, and recover when the system cannot help. A strong answer that appears in an unusable workflow may still fail the customer.

Check whether users understand what the system can do and what it cannot know. Give them a way to edit or reject a suggestion. If the product can change records or contact people, make the action and its consequence clear before execution.

Include difficult cases in your evaluation: incomplete notes, conflicting instructions, an unfamiliar situation, and a request the product should decline or escalate. A demonstration with one ideal input is not a meaningful reliability assessment.

When the first test disappoints

Separate failure of the problem, solution, implementation, and test. A participant may have a real need but find your prototype confusing. Or the prototype may work well for a problem that rarely matters. Those lead to different next steps.

If AI helped create the test, review whether its questions were leading. If participants praised the concept but avoided the next commitment, investigate the gap. If reviewers cannot explain why an output is wrong, improve the quality criteria before adding features.

The practical advantage of AI is more opportunities to learn within the same effort. Use that advantage to make uncertainty visible and decisions reversible. A smaller product with evidence behind its core behavior is more useful than a large roadmap built from plausible sentences.

Questions you might be asking

Can synthetic users replace customer interviews?

No. They can help explore scenarios or improve a research script, but generated responses are not observations of real people.

What should I prototype first?

Prototype the part that tests an important uncertainty, such as whether someone can complete a task or understand a decision. Avoid building the entire product before that question is answered.

How do I know whether an AI feature works?

Use realistic cases, explicit quality criteria, and human review of the actual workflow. Include correction effort, failures, and recovery rather than judging only the most impressive output.

Sources and further reading

• Microsoft human-AI interaction guidelines
• Anthropic evaluation guidance
• Microsoft appropriate reliance research

Sources checked October 16, 2026. Review the linked publisher information for current details.

Put one idea to work

Define the customer outcome, smallest useful deliverable, evidence, and review decision before expanding the work.

Plan an expertise pilot →Explore more from Outcome DrivenExplore the Outcome Club signup →