As our team is currently working on revamping, redesigning and polishing many of the services of Pathfinder, one of the major portals here in Greece, I caught myself wondering more than once…
When a user is presented with a rating form under an article, what does he usually rate? The actual article and the way it was written or the piece of information the article describes?
And what should he rate? How are these rating things supposed to work anyway?
Have you ever thought about that? It may sound trivial, but most of the times I thought I’ve reached a conclusion about it, users proved me wrong.
6 comments on this post
stelabouras #1
04.Mar.07
Let me add one more question, cause I am wondering lately (for a project that I am working on), “How” should he rate?
a) Classic 5 star system?
b) Digg-style thumbs?
I know that both are little…exploitable but I think the first classic way is better..
cosmix #2
04.Mar.07
Hey ya. It’s a pretty good question and most rating systems I’ve seen usually do not place firm and clear boundaries between form and content.
Typically, rating systems ask you to respond to the question: “Did you find this article useful?” or “How would you rate this article?” without explicitly specifying what they are referring to. You could, theoretically, apply some heuristics or randomly ask a small random sample of the users to rate specific aspects of the site (i.e. form, presentation, content, aesthetics, whatever), although I believe you could just take the ‘generic’ rating to mean — say — 70% content and 30% presentation. In other words, I believe that it is the content that is primarily being rated and to a lesser extent the presentation/expression of the actual text. It is, if you prefer, the overall perception of the article that is being rated and that combines all of the above aspects. Of course, content and presentation overlap to a considerable extent: really good content can be blemished by bad presentation and vice versa.
Some rating systems I’ve come across over the years break it down by default and ask for more detailed feedback, but I think you’ll find that most users will be turned away by multiple rating categories etc.
I’d say: stick with a single classic five star rating and provide a link to a feedback form (not a comment form) for typos, presentation errors etc.
Sugar #3
04.Mar.07
@ stelabouras: That’s the logically next question… My opinion is that one should stick to the 5-star system for anything content-heavy, while the yes/no system should be used in special occasions, like rate people responses to forum threads, articles, etc. More like, 5-star for content quality and yes/no for content usefulness.
@ cosmix: It’s a tricky question, isn’t it? But I agree with you, most of the times people rate content and not presentation, except for the occasions when the presentation is so bad, they rate that. Complicated?
Your idea seems to be a catch-all, thanks for sharing.
cosmix #4
04.Mar.07
You’re very welcome. Always glad to help when possible.
gterez #5
04.Mar.07
Users rate however they feel like, no matter what you urge them to rate, and this is especially true in the case of content as diverse as news articles on a portal: Positive feelings -> good rating, negative feelings -> bad rating. It’s really hard to influence the process and the criteria, no matter how well you design and present the rating mechanism and specify guidelines. And it is so because the very act of rating is urged by someone’s strong opinion on the particular piece: few people will give a rating of 3 (out of 5) to an article in a news portal. 1s and 5s will more often be the case. So, if a reader hates your article’s writing style, they’re likely to give you an 1 and get on to the next page, rather than contemplate about the rating criteria you have specified and give you a 4 on the content instead.
Article writers and other readers should perceive the rating in the same manner:
good rating -> “there’s something here, either interesting or nicely said or funny or just plain pretty”
bad rating -> “there’s something missing here… look elsewhere”.
I totally agree on the “5-star for content quality and yes/no for content usefulness” idea, but I should point out that you’d better only go for one of those schemes and not make it complicated by asking users to differentiate between the two.
It’s another one of those cases where computer guys feel the urge to ask the user more questions so as to be able to present him/her with richer, more insightful results. But the users want simple, usable, meaningful stuff. They don’t care about things like “where can I find some statistics about writing style vs. content value on this portal”, these interest only the site owners. Users will just click on a couple of pages that come up with good ratings and if they like what they see they’ll hopefully stick around your site and read some more, if not they’ll just look somewhere else. And simple rating schemes work because they’re just dead simple: no explanations necessary, just rate the way you want to.
Sugar #6
04.Mar.07
@ gterez: Some really good thoughts there. You’re right, users should not be presented with this dilemma.
Too many options presented to the users is not always a good solution.