Dan Ariely: 7 Problems with Research Groups and Surveys

It’s an oldie but it’s a goodie – an interview by Nex Gen Research with psychologist and behavioural economist Dan Ariey, author of Predictably Irrational (speed summary) and expert on non-rational (non value-maximising) consumer behaviour.

In the interview Dan suggests that market researchers are irrational – and predictably so – in their over reliance on traditional research groups and surveys that generate biased findings. Specifically he identifies 7 problems with research group and survey methodology.

1. Traditional survey or research groups that pay people for their opinions biases both who participates and what they say; the desire for cash and to please drives this bias
2. Research groups and surveys that solicit opinions from those without expert or extended experience generate stereotypical results that are biased by ignorance – “it’s not useful to ask [these] people their opinion”
3. Research groups an surveys that over reach from “what” questions to asking “why” questions create interpretive bias; the research wrongly assumes people have some privileged insight into why they do what they do (and they don’t)
4. Survey research (and worse research groups used as mini-surveys of people’s opinions) is but a quick, cheap and inferior substitute for a real experiment or field test that sacrifices validity, reliability and predictive power.
5. Focus group verbatims that pepper presentations can provide a dangerous, false, and erroneous sense of confidence in a business direction or decision. It is quite irrational to base business decisions on the uninformed, non-expert opinions of consumer groups ‘It sounds crazy but organizations do it all the time’
6. Traditional research methods such as research groups and surveys create an artificial environment that bias results because they are so very different to the natural real-world decision environments that strongly influence our thoughts and behaviours
7. Surveys used to segment people into groups create artificial differences – we are all much more similar than we think.  Good research should look for similarities not differences between people