25 become 50 – Our rating system is being revised

You are given an individual rating for every workitem you process on clickworker.com. Depending on how good or bad it turns out, it affects your rating as an author, translator or web researcher. We will be making a few changes in the rating system shortly. These changes are primarily designed to ensure that your ratings are less susceptible to extreme evaluations, and keep decreases moderate.

What will change? Two items.

First of all, we will double the number of orders that determine your rating: 25 become 50. This already reduces any possible changes in your rating per order. Therefore your rating value remains more stable on the whole.

In addition we have also added a further mechanism: by letting older evaluations flow into the new evaluation we can “smooth” out the overall result.

This is how it works: based on your current value you have a “standard zone” that is respectively 10% above and below your value. For an author with 85%, this would mean that the standard zone ranges somewhere between 75% and 95%.

If this author receives an evaluation for a new text, there are two options open: if the evaluation is within this standard zone, exactly this amount will flow into the rating.

However, if the author receives an evaluation that is outside of this range the new “smoothing mechanism” will come into effect. Instead of the deviating evaluation, an average of the last three evaluations will flow into the value rating.

It might initially sound a bit complicated, but the following sample calculation will help illustrate how it works.

Let’s assume that the last two texts written by our exemplary author received 80% and 85%, ratings that are generally within his range. He had a bad day though today and he only got 50% for his next text. This would normally signify a drastic decrease in his rating. But the system now determines that this evaluation is outside of his “standard zone”. Instead of letting the 50% flow directly into the rating, and average value is calculated using this, and the last two orders. In our example this would amount to approximately 72%.

And these 72% – instead of the actual 50 % – are exactly the amount that will flow into the statistics for this text. The previous good work compensates for the bad evaluation and the rating of our exemplary author does not fall as sharply as before.

7 Kommentare