Contemplation: Scale Ratings and Why They Suck

“It’s easy to know if a video (or anything) is good or bad, but how on Earth do you determine if it’s 2 star, 3 star, or 4 star-worthy? Everyone likely has their own opinions about what would constitute those ratings, and naturally, they’re all completely subjective.”

~MG Siegler, from This TechCrunch Post

So the above quote (and the full blog post from whence it came) entirely summarizes my loathing of rating scales.  Simply put: BECAUSE THEY ARE FUCKING ARBITRARY AND DON”T TELL ANYBODY SHIT! I have had, and still am having, personal experience with the annoyance of rating scales.  In work we use rating scales on evaluation forms, during applicant interviews, for patron surveys, etc. and I can’t stand any of them. 

It all comes down to the fact that there is no way to ensure a consensus on what the numbers (or letters) on the scale represent. Say you have a scale of rating something, say quality of a training, and the scale runs from 1 – 5 with 1 = no quality and 5 =great quality and then everything in between (4 = good quality, 3 = average quality, 2 = little quality).  The problem is, even with the numbers having values spelled out, determining what the difference between “average” and “good” quality really is.  As Mr. Siegler keenly points out, it is all subjective and depending on how people read the scale or interpret its meaning then there is little that can come out from it.  The only two that really end up mattering are if it was a 5 (Great!) of a 1 (Shitty!).

So what is the solution?  How do we get feedback about what is good and what is bad?

Simple: Ask yes/no questions and solicit feedback.  Is this a 100%  guaranteed method for getting good feedback or providing good information.  Of course not, but what it does is allows a person to answer simple questions and then give comments to explain their answer choice.  So for the above, instead of “Rate the quality of the training on a scale of 1 -5” have the question read “Was this a quality training?  Please explain your answer.”  With this yes/no you have a much better chance of finding out what you wanted to know from the get-go (that being whether or not the training was quality).  Now sure, there is the risk that a good comment explaining the answer choice might not be provided, but in that case you just go with it.  However, hopefully, where good choice explanations are provided you now have some feedback to work with in the future.

Really it is all about semantic and the way in which we interact with language.  We use scale systems with the assumption that they are giving back to us useable and valuable information, but often overlook that there is always a need or interpretation.  If we solicit feed back where somebody can answer a question and provide explanations we are then finding out the information outside of an arbitrary system.

Now of course, in the day and age of computers and the Internet, there are statistical values to a numbered scale system, because we can see how people respond.  Guess what though?  You can do that with yes/no response too!  Sure to pull data out of a written response you may need a human mind to sort through the responses and find like comments/concerns, but “yes” and “no” can be recorded just as easily as a 1 or a 5.

The other end of the challenge is making sure that we know what it is we want ratings on and why we want these ratings.  For something like YouTube, in Mr. Siegler’s post, the rating are obviously intended to help determine popularity of the video (probably along with the number of actual viewings).  But here is the thing, doesn’t simply asking “Did you like this video?” provide a much more clear picture of whether or not there is a popular response to the video?  YouTube already allows commenting and you can be damn well sure that people on the Internet are going to provide feedback when there is a commenting feature available.  A yes/no answer to whether a video was enjoyed very clearly defines whether the majority of people like it or not.  On the other hand the star system (a 1-5 scale) simply tells that people have chosen some degree of stars, and like Mr. Siegler pointed, the people most likely to rate with stars are probably the ones who like the video in the first place.  You can see a similar disproportion of high ratings on for just about any one of their products. 

So just a little something to think about before the next time you create an evaluation or survey or what have you (or you partake in one yourself).  Why be arbitrary when you can be straight forward and honest?

~ by Nathaniel on September 23, 2009.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: