Is it a good idea to ask people to rate products all at once?
I’ve just finished working on a project where we asked people how likely they would be to purchase a number of different products. However I was worried—not losing-sleep worried, just do-a-bit-of-thinking worried—about how rating all the products together, rather than separately, would affect the results of my purchase question. I’ve done some digging without developing any concrete answers, but came across an interesting concept…
The evaluability hypothesis
The evaluability hypothesis was introduced by psychologist Christopher Hsee (1996) and is stated as follows:
When two stimulus options involve a trade-off between a hard-to-evaluate attribute and a easy-to-evaluate attribute, the hard-to-evaluate attribute has a lesser impact in a separate evaluation than in a joint evaluation, and the easy-to-evaluate-attribute has a greater impact.
In plain English: when you show people things one at a time, they fixate on whatever’s easy to judge. Show them things side by side, and suddenly they notice the stuff that’s harder to evaluate on its own.
What on earth does that mean?
The evaluability hypothesis breaks (product) attributes down into two types - easy-to-evaluate independently, and hard-to-evaluate-independently. Easy-to-evaluate attributes can be assessed by a person without needing to compare it to something else. Hard-to-evaluate attributes are difficult to assess without comparing them to something else.
If you have products which have an easy-to-evaluate attribute and a hard-to-evaluate attribute, when the products are evaluated independently people will place more emphasis on the easy-to-evaluate attribute. If the products are evaluated togetherthen people will place more emphasis on the hard-to-evaluate attribute.
Seriously, what are you going on about?
An example might help explain where this is leading! Imagine were doing some research which involves looking at how much people are willing to pay for two cameras. We ask people how much they would be willing to pay for each of the cameras assuming they had a budget of 10 - 150 to spend on a camera.
| Camera | Megapixels | Defects |
|---|---|---|
| 1 | 5 | None |
| 2 | 10.2 | Scuff on the case otherwise as new |
If the cameras were rated separately, you’d expect Camera 1 to get the highest mean value (in terms of how much people would be willing to pay) because people may not be sure how good 5 mega-pixels is (hard-to-evaluate attribute). But they know no defects is good (easy-to-evaluate attribute). If the cameras where rated together you would expect Camera 2 to get the higher rating because people know no defects is good (easy-to-evaluate attribute) and that in comparison 10.2 mega-pixels is better than 5 mega-pixels (hard-to-evaluate attribute).
So, what does it all mean?
So what does this mean for research design? If your product has a hard-to-evaluate advantage (like megapixels, or battery life, or some technical spec), showing it alongside competitors might help people appreciate it. If your advantage is easy to evaluate (no defects, lower price, familiar brand), maybe you’re better off letting it stand alone.
I still don’t have a definitive answer to my original question about rating products together versus separately. But at least now I have a framework for thinking about when it might matter.
References
Christopher K. Hsee, The Evaluability Hypothesis: An Explanation for Preference Reversals between Joint and Separate Evaluations of Alternatives, Organizational Behavior and Human Decision Processes, Volume 67, Issue 3, September 1996, Pages 247-257, ISSN 0749-5978, DOI: 10.1006/obhd.1996.0077.