How we think about accuracy in synthetic research
Accuracy depends on the question. We describe outputs as experimental, publish limits openly and recommend treating strong signals as a reason to run a human test.
Any tool that claims one accuracy number for every question is hiding something. How well a model predicts depends on the question, the audience and how much is known about both.
What we say instead
- Results are experimental model output
- They are best at ranking options and finding objections
- They are not a census, a forecast or a substitute for field research
A rule of thumb for acting on a result
- Is the decision reversible? If so, act on a strong signal.
- Does the result hold when you rerun it with a different brief wording?
- Do the reasons make sense when you read individual personas?
- Would five real conversations confirm it?
If you cannot say yes to the last question cheaply, the synthetic result is a hypothesis, not a finding.