Can you trust AI-generated customers with your next business decision?
Can you trust AI-generated customers with your next business decision?
By Alex Mercer, Evidence & Insight Editor, Expert View Media
Imagine a supplier offering feedback from a thousand potential buyers before your next planning meeting. You can test a proposition, revise the wording and ask another question without recruiting anyone.
The catch: the answers come from AI-generated customers.
For a business weighing a new price or product launch, the appeal is understandable. The harder question is what those answers are good enough to support.
Synthetic research deserves a careful hearing. Buyers also deserve a clear account of what has been tested, where it fails and how those failures might affect their decisions.
Why this conversation matters now
The commercial interest is visible. A Greenbook webinar advertised for 15 October features Google Cloud and Focus discussing digital twins in customer research, including messaging and strategic planning. Focus sponsors the session. Its description offers a useful indication of the applications being promoted, but does not independently establish their effectiveness. Greenbook’s event listing.
Alongside that interest, recent research raises questions about reliability.
On 30 September, Pew Research Center published an experiment asking AI to answer surveys as real members of its American Trends Panel. The model received information about each person, including demographics and previous survey responses.
Across three replicated survey waves, the average absolute difference between synthetic and human results was 12.4 percentage points. The headline analysis used Claude Opus 4.6, and performance varied between groups and questions. Pew Research Center’s findings.
That is a substantial warning, with a specific scope. Pew tested US public-opinion surveys under particular conditions. Its results cannot be transferred wholesale to every commercial platform, specialist business audience or research task.
For a buyer, the implication is to request validation relevant to the proposed use. A demonstration that works elsewhere is a starting point for questions.
Establish what you are buying
“AI research” can describe very different activities. Summarising interviews with real customers and generating simulated customer answers have different relationships to evidence. Buyers should establish which activity produced each finding in a report.
A September preprint by Oded Netzer and Rajan Sambandam distinguishes ungrounded AI responses, personas representing customer segments, and digital twins built around individual people. It also warns that good agreement with population averages can conceal poor differentiation between individuals. The authors’ paper.
That distinction matters commercially. A system might reproduce the broad appeal of a familiar product while offering little dependable guidance about which customers would choose it.
The paper explores selective augmentation of existing research. It is a preprint, and one author is president of research company TRC Insights; readers should consider its findings with that status and industry connection in view.
The useful purchasing question is therefore quite precise: what has this particular system demonstrated, for this audience and this kind of decision?
Ask whether an error would change the recommendation
Consider a hypothetical software business comparing two messages: saving staff time and reducing operational risk.
Its synthetic panel slightly favours time savings overall. The business plans to use that message across its next campaign.
Now suppose the tool represents everyday users reasonably well but misses the concerns of people approving the expenditure. An apparently modest weakness could matter greatly if the campaign needs to persuade those budget holders.
This is an illustration, not a reported case. It shows why a single accuracy score leaves important questions unanswered. The team needs to know whose answers are reliable and whether the remaining uncertainty could reverse its choice.
The same reasoning applies to price. A tool might rank two propositions correctly while giving a poor estimate of willingness to pay. Those results would support different decisions.
The most useful evidence shows whether the tool’s errors would change what you do.
Before commissioning research, write down the intended action. Choosing which ideas deserve further investigation carries a different burden of proof from setting a launch price or abandoning a market.
Five questions for a synthetic-research supplier
1. What real customer evidence shaped these responses?
Ask where the underlying information came from, how recently it was collected and which relevant buyers it covers. In a B2B study, specify the roles involved in the purchase. A credible simulation of a product user does not automatically represent procurement or the budget holder.
2. How did you test it against answers it had not already seen?
Request a comparison with relevant human responses withheld from the system’s construction and tuning. Ask whether the evaluation resembles your proposed task. A convincing demonstration is less useful when the information needed to reproduce the answer was already supplied.
3. Where did it perform poorly?
Request unsuccessful questions, weaker customer groups and examples where the recommendation differed from the human benchmark. Evidence becomes easier to assess when the boundaries are visible.
4. How much uncertainty could our decision tolerate?
Agree this before viewing the results. If a small difference would determine which product receives investment, ask how confidently the method can distinguish those options. Include the limitations of the human benchmark in that discussion.
5. What triggers fresh research?
Ask what happens when the model changes, the market shifts or a question falls outside the validated use. Establish who decides that new customer interviews or surveys are needed, and include that work in the cost comparison.
Give simulation a defined role
A sensible starting point is a bounded trial. Choose a question, specify the decision it could inform and agree how the synthetic answers will be checked before allowing them to influence a larger commitment.
Evaluate the work saved alongside the work added: preparation, validation, interpretation and any investigation needed when results disagree. A lower initial research bill is only part of the commercial calculation.
Keep the evidence trail visible. Reports should identify which findings came from people, which were generated and which conclusions are the analyst’s interpretation. Colleagues using the findings later need that distinction as much as the original research team.
Synthetic research may earn a useful place in a business’s decision process. The next supplier conversation should make that place concrete: show us a relevant test, show us the failures, and explain what remains uncertain before we spend.
Comments