I’ve long said that the rating system in Protagonize is inherently flawed because it relies on the honesty of it’s participants, both in that it relies on users not artificially inflating their own ratings and on user not artificially deflating the ratings of their peers. As a moderator on the site and as a coder who has a passing interest in reputation systems, it’s clear that relying on user’s honesty isn’t a workable solution.
The other problems with rating systems is that there tends to be a large bias towards positive ratings. In Protagonize, where the people behind the ratings are anonymous to other users, there is a clear trend of ratings towards the 3.5-5.0 end of the scale out of a 0.5-5.0 range. Looking across the site, it’s clear that many things rated highly do not deserve these ratings and in an ideal world, the range of values would see a concentration of works rated around the 2.5 mark, with outliers at the more extreme ends of the scale. However, this doesn’t seem to be the case, though without access to the actual statistics I can only make this assertion based on my own observations, which will be prone to their own biases.
From what I have observed, the majority of users on protagonize behave in the following way. Upon reading a work, if they like it, the will rate it highly, at a minimum 3.5. If they are friend with the author, they tend to naturally increase their rating slightly, making a 4.0 become 4.5, for example. 5.0 tend to be given out slightly less often, being reserved for either friends or for exceptional pieces of work. If on the other hand user dislike a work, they tend to abstain completely from rating a work, most probably because they don’t want to hurt the author’s feelings, feel like they are giving the author a chance to improve before they rate them accordingly or just can’t bear to have to think about the work any longer. Obviously this creates a strong bias towards high ratings, as low ratings tend to be a minority.
Nick, the founder of protagonize, recently posted on the protagonize blog about the misuse of the term ‘hater-rater’. The terminology was originally aimed at people that deliberately try to warp the system by actively negatively rating people, usually those with high-ratings, in order to both lower those authors rankings whilst increasing their own relative to said authors, since they are essentially dragging the site-wide average down, putting their lower scores closer to the upper bounds above the average than the lower bounds. Over the course of time, the term has come to encompass anyone who makes any kind of negative rating, or people who rate lowly due to ‘not getting’ a piece of work, rather than any objective measure. Being branded a ‘hater-rater’ is obviously not something people want, which further leads people to instead abstain entirely from rating where they otherwise would have given a low rating. As such this leads to the above bias towards ratings that fall within the 3.5-5.0 range and I can see this perpetually refining itself as the new lower bound gets considered hateful, slowly narrowing the range tighter and tighter until rating become 5.0 or nothing at all.
These problems are only exacerbated by people who write to achieve high ratings, rather than writing well and being rewarded with high ratings. Since your overall rating is calculated as an average of all your ratings, it makes sense, when trying to competitively climb the ladder, to write a much higher quantity of posts targeted at a group of friends you know will rate highly, rather than at actually trying to write well. This way, you can both raise your average rank whilst also building in increasing immunity to low ratings, by virtue of a overwhelming majority of high ratings.
Seeing this problem some time ago, I wrote The Protagonizer’s Manifesto, a call to action for fellow authors to rate everything they read, all the time, and to rate them objectively, leaving their emotions and personal opinions for comments, which are far more suitable for relaying that kind of content than a single number. While I had many people agree with much of what I said, nothing much has changed. I’ve proposed a few other systems for ratings, but the core problems remain as part of any rating system. Namely:
- People abstaining from rating
- Only rating things positively
Now, for a non-binary ratings system, both of the above pose a problem and lead to the biasing problems seen above. Facebook solves these issues by actually making use of abstinence as a form of negative rating. By allowing users to ‘Like’ something but not providing a way to dislike anything, nor a way to provide a more scalar value of how much one ‘likes’ something, they’ve eliminated both problems. Things that have more likes are better than things with less likes – that’s a clear and obvious metric.
However, for protagonize, does a purely cumulative numeric metric actually provide a useful function? Let us first examine what functions ratings currently serve on the site.
- Determining approval of an individual page of work
- Determining the overall approval of an entire work
- The visibility of that work on the site in terms of recent approval
- The visibility of that work in terms of all-time approval
- The visibility of the author in terms of recent approval
- The visibility of the author in terms of all-time approval
- The average opinion of the author in terms of ratings given by said author
Does a purely binary system of rating still achieve these goals and further more, are these goals desirable? Using a purely cumulative system, the first six features are all met. Is the last one important, given this new model? Perhaps not, since opinion has been completely removed from the equation, as things are either good or not-good, so it can’t be said any author has an overall opinion but only that an author has liked a certain number of works. However, are those first six points desirable, what exactly do the serve in terms of usefulness to other users?
For the most part, one would expect the visibility on the site to be a kind of endorsement or recommendation – an increased visibility implying one should read the work whilst a decreased visibility implying it is one to avoid. The problem with that assumption is that it has no context and provides no information about a work other than a certain number of other people ‘liked’ it. As an author, you don’t know why people liked something, which is not very helpful, while the existing system at least attaches some context to each of the scalar values it provides in terms of ‘Perfect in every way’, ‘Lack originality or suffers from serious mistakes’, etc. As a reader, all you know is that other readers liked this, but not why and as such you can only judge something on it’s perceived popularity rather than whether or not it meet’s your own criteria for what you consider good.
Thus, in terms of recommendation, just listing things or increasing their visibility based on the total number of ‘likes’, for protagonize, seems to be little more than an impetus for those people who want to be at the top of a list, rather than actually any good, which promotes rating abuse such as fake accounts, liking friends work, reciprocal ‘liking’, etc. However, there are ways of dealing with those kinds of issues and there have been many papers on the subject, such as ‘Immunizing Online Reputation Reporting Systems Against Unfair Ratings and Discriminatory Behavior’. Most recommendation systems are based on correlations between you as some who rates things and others who also rate things, building a set of of items commonly rated by others that share many similar ratings to you, but for which items you have not yet rated. Something like this is perfectly applicable to protagonize and far more useful in terms of recommending things you may want to read than lists based ‘likes’ or the current ratings, but such system fall outside the scope of this.
Ratings in the current context of protagonize are markers solely of personal reputation. It can be said that those with a higher reputation are better writers, or at least the implication is there, but due to rater bias and the increasing marginalisation of ratings to within such a narrow boundary, such reputations lose their meaning. Switching to a binary system doesn’t change this fact as the problems of gaming the system still arise, changing the issue from collecting high ratings to one of collecting ratings at all. So in terms of personal reputation, perhaps a non-numeric value is better placed for presenting this. Let us get to the heart of the matter.
What qualities make a writer a good writer? What qualities make a work good or bad? Many of these things aren’t quantifiable or otherwise require intense computation to work out automatically. As such, trying to map this to any numeric system isn’t going to work. What’s needed is a more flexible system based on describing qualities of a work, rather than rating it good or bad.
I’ve proposed a system like this before, based on tagging inspired by that which occurs in LittleBigPlanet on the PlayStation 3. In LBP you could choose from a series of adjectives to describe a work. I propose extending this format significantly to add the following features:
- Positive, neutral and negative connotations
- A system for indicating the reliability of any rater
- Increased context for enabling feature recommendation systems
In this system, ratings would be given via three boxes. A positive, neutral and negative box. Into each box, you could choose several tags from a pool of attributes and drag them into whatever box you liked. For example, you might indicate you felt positively about the storyline by dragging the storyline tag into the positive box, while you might indicate you felt indifferent to the setting by dragging the setting tag into the neutral box and indicate your displeasure at the writer’s grammar by dragging grammar into the negative box. This first step of rating gives both the rater and the ratee more context into what exactly is being rated and what those opinions mean. Things that are not worthy of note need not be dragged into any box, essentially abstaining for those tags, but with a rich corpus of descriptors, abstaining becomes less likely. In terms of rating the reliability of raters, raters can be ranked based on how works they have rated have been rated by others. When one rater chooses to place grammar in the negative box while several others put the tag in the positive box, in can be said the distance between that rater is less reliable, having an outlying value rather than approaching the norm. From the distances these outliers have from the normal value, a reliability can be calculated and be used to scale the effect any outlying rating they have given accordingly. In terms of displaying these ratings so that they are useful, it would be encouraged to use several synonyms of various basic descriptors to avoid copying existing ratings or introducing bias when raters are choosing what ratings to give. For authors wishing to review their feedback, a tag cloud for each of the positive and negative values could be used, or perhaps just a list of the first few most popular tags (taking into account synonyms) for each of the positive, neutral and negative aspects. Lastly, the additional context these tags lend to recommendation system is obvious. Raters inclined to rate things positively in terms of grammar, imply a preference for things with good grammar, users using the sci-fi in a negative context a lot of the time can be said to dislike sci-fi. The benefit of using nouns for descriptors is that each has no emotional context except that which is given by selecting what box to put it in. As such, a author seeing sci-fi high up in their negative box need not feel badly, for it’s clear that many people who don’t like sci-fi have read the story, rather than there being anything wrong with it per se.
Other ways of presenting a simple binary or trinary system with additional helpful context is to follow a method similar to getsatisfaction, where you state your mood (good, bad, neutral) and are then offered a way to contextualize that by adding tags or emotions to the rating. Such a system could work in protagonize where you choose good and then the option expands to let you choose what things from a list of items specifically made it ‘good’ or an option to add your own message while still maintaining your anonymity.
The core issue is that scalar rating systems suck for subjective assessments. Basing a system around any kind of scalar rating is inherently flawed for a site like protagonize where everything is highly opinion based, or is quantifiable, but only by human beings and then only in a fuzzy way. Spelling, punctuation and grammar for example are all quantitative in terms of it being good or bad, correct words versus incorrect words, etc but a machine isn’t capable (yet) of accurately coping with ratings these and putting such ratings in the hands of humans is prone to error and bias. No, the only option is to get rid of a ranking system entirely, and instead encourage others to judge each other qualitatively and to punish those that try to game the system by unilaterally effecting the worth of all of their judgements, benefiting everyone.
From that basis, it’s possibly to construct a quantitative ranking system based upon the number well-supported, agreed upon qualitative statements, weighted accordingly for the subject domain, which restores the previous listings, but in a much fairer manner, with additional context that makes such things much more useful for everyone involved.
Like this:
Like Loading...
roclafamilia, and 1 other are discussing. 
Hey Andrew,
First off, let me thank you for posting this. Taking the time to consider a core system of the site so thoroughly is much appreciated. I do recall you bringing up a similar system previously, but since we’ve got it all down on paper, so-to-speak, let’s hash it out a little further.
Second, the system you suggest is actually very close to something I’ve developing for another site I’m working on. It’s funny that it parallels it nearly identically (without the nouns, but using descriptors/qualitative commentary, depending on an underlying -1/0/+1 system.) Not quite the same implementation, but surprisingly close. Great minds…?
Maybe I should give you a preview…
I do have a few issues with the system you suggest as it applies to Protagonize, though. There are a couple of things that your suggested system doesn’t take into account that might cause (me) major pain in attempting to convert over. While I do really like the guts of the idea (particularly how this could impact dynamic recommendations down the road), these issues would need to be resolved before I could even consider implementing a new system.
I’ll save a more in-depth discussion of these points for our beta forum (because I’d really like to hash them out more, and I’ll start a discussion there as well), but I’ll briefly mention my most pertinent concerns here:
(1) How does this impact historical ratings?
Obviously, one approach could be to wipe them altogether. Although I think that would likely end up with me hanging from my neck at the nearest tree branch, surrounded by angry Protagonizers with pitchforks.
You’re likely aware that I’ve done a lot of research into ratings systems (hence the first change to half-marks), and I have been considering switching to a Reddit-like up/neutral/down (effectively +1/0/-1) system for a long time. Migrating to that system would be easy enough based on existing ratings, but migrating to a combined system like the one you suggest would be problematic. The only way I can see it working is to create a base initial rating that migrates our existing /5 ratings to a +1/0/-1 system, but with no descriptors selected to start. After that, future ratings would include the descriptors as well.
(2) How would “Top Rated” lists around the site function?
Would these be junked altogether and replaced with something else?
The “Popular” list would likely be unaffected in that it is based on a number of aggregated factors, but our Top Rated lists (which I would be happy to remove) would need to replaced with something as-or-more valuable to authors on the site.
Any suggestions on how that would work?
My concern here is how aggregated ratings would show up, not only with respect to newly-rated material, but with respect to existing works as well that have been migrated to the new system.
Which leads me to…
(3) How are author rankings calculated in this system, if at all?
A problem I can see right away is that if 100 of your “negative” ratings comprise of people who dislike sci-fi, even though you happen to be an excellent sci-fi writer, you will still end up with a horrendous overall rating.
Obviously, the vocabulary selected for rating would impact this tremendously, and having subjective options like “sci-fi” (using this since you used it as an example) in addition to more objective descriptors like “grammar” or “plot” seems like it would make life a lot more difficult. I can see how this could be very valuable to generating recommendations, but I can’t see how it would work, realistically, given that attaching personal taste to numeric ratings could have a major negative impact on someone’s overall rating.
There are probably other issues as well, but that’s a start. I like where you’re going with it, and I’d like to see if it could evolve into a system that could replace the current mechanism. Just needs a little more refinement.
Cheers,
-nick / protagonize