Artixol
Method

How we think about accuracy in synthetic research

TL;DR

Accuracy depends on the question. We describe outputs as experimental, publish limits openly and recommend treating strong signals as a reason to run a human test.

Any tool that claims one accuracy number for every question is hiding something. How well a model predicts depends on the question, the audience and how much is known about both.

What we say instead

  • Results are experimental model output
  • They are best at ranking options and finding objections
  • They are not a census, a forecast or a substitute for field research

A rule of thumb for acting on a result

  1. Is the decision reversible? If so, act on a strong signal.
  2. Does the result hold when you rerun it with a different brief wording?
  3. Do the reasons make sense when you read individual personas?
  4. Would five real conversations confirm it?

If you cannot say yes to the last question cheaply, the synthetic result is a hypothesis, not a finding.

Bring a question. We will staff the audience.

Tell us who you need to understand. We will build the audience and run a study with you.