XKCD on Online Star Ratings
I think I may have written on this subject before, but I am too lazy to look it up right now, so I will just express my point again (you can deal with that right? No? Okay, well screw you, go away).
Over the past several years I have been tasked with creating a number of surveys and evaluations to gather feedback and information on all sorts of matters. I am not formally educated in quality survey creation, though I did take a couple of years of statistics classes and have read some articles and books on good surveying skills. Basically though, these opinions are my own and not really an account of real scientific evidence. Anyways, one conclusion that I have repeatedly come to, is that questions that rely on some scale system where people are asked to indicate their response to something between a number of variable (usually at least five options, sometimes more), are not very helpful at letting us know what people think. Basically people who mark the extreme on either end of the scale (e.g. the one star or the five stars), are the best indicators, while everything else becomes a bit of fluff. Now, admittedly, with star review systems, I think it is well agree that four or more stars equals good, anything less sucks. But, ideally this shouldn’t be the case. Three stars should represent an average, as not great but also not horrible.
Why does this happen? I think there might be a number of reasons. First of all, I think providing too many options (and for the record I think five options is too many), creates a degree of arbitrary certainty in what one ranks something. You end up asking yourself “Does this really deserve a 5 or is it really just a 4? Or maybe it is a 4.5?” While some people will have certainty, many won’t. What makes something earn a 2 instead of a 1? I know we tend to think that this is logical, and shouldn’t be hard, but in truth I think it is very hard. Additionally, if two people don’t agree on what two actually means, but they both rank something a 2, then there is a discrepancy.
My personal belief, is that the best kind of feedback, is when you give people no more than three options, though preferably two. basically those two can always be boiled down to a “yes” or a “no” (with the optional “maybe” or “neither” if you are really a stickler). Basically the feedback you get from these two (three) options lets you know whether something is liked or not, or if you disagree or not (rather than how much you disagree). Add to that, an option to provide some written comment and feedback (specifically tied to the “yes” or “no” answer), and then you are getting a much clearer picture of people’s opinions, and you’ve removed more options for arbitrary interpretation.
Anyways, as always, the genius of XKCD leaves me smiling this morning. Hope y’all have a good Wednesday.
General Lordisimo's Apocalypse 