Synthetic panels get the average right and miss where demand shifts first

Synthetic respondents are failing their first serious tests. On 16 October 2026, Nielsen Norman Group showed AI-generated panels reach the human conclusion but lose the rare answers. An updated benchmark shows they also invent segment differences. That hits concept tests, segmentation, targeting and the evidence behind budget requests.

What did Nielsen Norman Group actually find?

The synthetic panel got the headline right and the detail wrong. NN/G compared 120 synthetic respondents from the Synthetic Users platform with an earlier survey of 604 tech and design professionals. Both groups answered the same open-ended question. Both reached the same overall conclusion.

The differences sat underneath. 84% of synthetic respondents used UX terminology, against about 70% of humans, a statistically significant gap. Humans used more than 50 distinct labels. The synthetic panel collapsed onto about a dozen. It produced none of the unusual or specialised terms in the human data. NN/G calls this mode collapse: the model repeats the most common answer and drops the long tail.

The skew even ran against the sample. Only 20% of the synthetic panel were UX professionals, against 67% of the humans. Yet the synthetic data leaned harder toward the dominant answer. 93% of synthetic product managers used UX terminology, against 44% of human product managers. The persona changed. The answer did not.

NN/G flags one confounder. The platform sometimes reworded the question, and researchers had limited control over presentation. Its conclusion is clear. Synthetic data covers the most common response at best. Less common responses and emerging themes require human participants. Synthetic findings are a starting point, not evidence.

Why do synthetic panels point to the wrong segment?

Synthetic panels confuse demographics with attitudes. The arXiv preprint When Synthetic Users Fail (v2, 4 October 2026) tested four LLMs from two model families. It scored them against General Social Survey and World Values Survey data. At the individual level, no LLM beat the strongest non-LLM baseline.

The segment results matter more for marketers. The models treat demographics as far more predictive of attitudes than they are among real people. On a segment-targeting task, they inflated between-segment gaps two- to fourfold. They pointed to the wrong segment in half of US cases and most cross-cultural cases. They created segment splits that do not exist. Larger models did not fix this.

The mechanism is simple. A synthetic respondent answers as the model expects someone with that profile to answer. That is a stereotype with a sample size. A segmentation built on it looks sharper than the market is. The sharpness is the error. Targeting, media allocation and concept selection then follow a difference real customers never showed.

Why does the long tail matter more than the average?

Demand shifts start in the tail, not the consensus. A synthetic respondent is the model's guess at what a person would say. That guess is built from past consensus. It is opinion at scale, not observed behaviour.

Consensus describes what the market already agrees on. Your competitors see it too. The advantage sits elsewhere: in the unusual phrasing, the niche need, the small group asking a new question. That is where a category starts to move, months before sales confirm it. The market moves first, and it moves at the edges.

NN/G's human panel produced more than 50 labels, including specialised terms. The synthetic panel produced about a dozen and none of the rare ones. Translate that to a concept test. The synthetic panel confirms the obvious idea and silences the early adopters of the next one. A method that removes the tail removes the earliest signal of change.

Where do synthetic respondents still earn their place?

Synthetic respondents are a rehearsal tool. The evidence base is large but immature. Synthetic Market Research 101: State of Industry, a compendium updated on 10 October 2026, counts 502 published works on synthetic respondents, 272 of them directly relevant. It states that guidance for market research is still developing. Consumer digital twins have less independent evidence.

Standards bodies are moving. In May 2026, AAPOR released Responsible AI Integration in Survey Research. The framework evaluates AI use in surveys on validity, reliability, sensitivity and performance. It recommends disclosure standards and sets up a new AI Subcommittee.

Practitioners draw the same line. The Alchemic, in 10 Market Research Trends Shaping Strategy in 2026, names two defensible uses: pretesting questionnaires and rehearsing analysis. It rules out segment sizing, because between-group differences are exactly what the models distort. That gives a working split:

• Use synthetic panels to sharpen questions, stress-test discussion guides and rehearse analysis before fieldwork.
• Do not use them to size segments, choose targets or justify budget.

What should leaders check before money moves?

Every synthetic finding is a hypothesis until real behaviour confirms it. Confidence without evidence is just another assumption. Three checks belong in every review where a synthetic result supports a decision.

1. Does it hold outside your own data? Your own customers show your own reality. Test the finding against what the wider market does.
2. Does the segment gap survive real behaviour? Assume synthetic gaps are inflated until observed data shows otherwise.
3. Where is the tail? If a study returned no unusual answers, the method hid them. Look for them elsewhere.

Observed behaviour offers what a synthetic panel cannot. How people search, compare and ask across Search, Social and AI shows demand forming before it becomes sales. That record includes the tail that synthetic panels erase. Trendata reads demand there: in what people actually move toward, not in what a model expects them to say.

A synthetic panel tells you what a model believes people think. A budget should follow what people actually do.

Frequently asked questions

Are synthetic respondents reliable for market research?

Only for the most common answer. Nielsen Norman Group found synthetic panels matched the human conclusion but lost rare answers. A cross-domain benchmark found no LLM beat the strongest non-LLM baseline at the individual level. Treat synthetic findings as hypotheses, not evidence.

Can synthetic respondents be used for segmentation or targeting?

No. The benchmark 'When Synthetic Users Fail' found models inflated between-segment gaps two- to fourfold, picked the wrong segment in half of US cases and invented splits that do not exist. Larger models did not fix it.

What are the defensible uses of synthetic respondents?

Pretesting questionnaires, stress-testing discussion guides and rehearsing analysis before fieldwork. Any finding that drives targeting, segment sizing or budget needs validation against observed market behaviour first.

Sources

• Synthetic Users vs. Human Respondents: Same Conclusion, Less Variation (Nielsen Norman Group, 16 Oct 2026)
• When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses (arXiv 2607.26348v2, 4 Oct 2026)
• Synthetic Market Research 101: State of Industry (updated 10 Oct 2026)
• AAPOR Releases New Report from Task Force on Responsible AI Integration in Survey Research
• 10 Market Research Trends Shaping Strategy in 2026 (The Alchemic)