Happy Holidays: A few adjustments to the ratings system

Hello everyone!

I hope you are safe and healthy and do your part to help reduce the impact of the pandemic until the nightmare is over. Stay home, if you must or can, be mindful and civil of each other and do not be an idiot. :)

For many parts of the world, today (or tomorrow) is a day of presents. So to celebrate the occasion I have brought with me a couple of changes and dare-I-say improvements to the ratings system. These are based on observations of the different ratings patterns and behaviours since we introduced the ability to rate (almost) every match in the database about half a year ago.

[These changes will be implemented and rolled out within the next couple of days. So if you do not see a difference immediately, just wait a bit. Celebrate in the meantime, if you want and can.]


1. Change to the Total Rating Calculation Algorithm
The total rating is the rating you see prominently on each database entry's right side. It is also the only rating used for all tables and listings. It is not the average of the user ratings, primarily to normalize items with less votes. For over 12 years, the algorithm used to calculate the rating was basically:

sum of all ratings / (number of votes + 1)

This worked well for our purpose -- mostly. The purpose was to normalize items with less votes, but the main effect was that it brought the total rating down. This means that it exclusively normalized "over-rating" and positive ratings, but not the other way. "Under-rating" and negative ratings were not impacted, and I intend to change this with a subtle change to the algorithm:

(sum of all ratings + 6.0) / (number of votes + 1)

Why 6.0? Because while 5.0 is the average of 0-10, the reality is that the overall average of all ratings of the side is more like 6 or 7, because most users tend towards rating more stuff positively. I guess that coloring of the ratings has a psychological affect as well -- I would love to hear from a psychology master about this at some point -- but mostly I believe the reason is that people mainly only bother to rate stuff they like instead of stuff they just find 'meh'. So I am using 6.0.

What's the effect? Well, consider the following examples.

                          |     Old Algorithm    |       New Algorithm       | New-Old | Old-Avg | New-Avg
--------------------------+----------------------+---------------------------+---------+---------+---------
  5 ratings of 10 points  |   50/(  5+1) = 8.33  |  (  50+6)/(  5+1) = 9.33  |  +1.00  |  -1.67  |  -0.67
  5 ratings of  9 points  |   45/(  5+1) = 7.50  |  (  45+6)/(  5+1) = 8.50  |  +1.00  |  -1.50  |  -0.50
  5 ratings of  8 points  |   40/(  5+1) = 6.67  |  (  40+6)/(  5+1) = 7.67  |  +1.00  |  -1.33  |  -0.33
  5 ratings of  7 points  |   35/(  5+1) = 5.83  |  (  35+6)/(  5+1) = 6.83  |  +1.00  |  -1.17  |  -0.17
  5 ratings of  6 points  |   30/(  5+1) = 5.00  |  (  30+6)/(  5+1) = 6.00  |  +1.00  |  -1.00  |   0.00
  5 ratings of  5 points  |   25/(  5+1) = 4.17  |  (  25+6)/(  5+1) = 5.17  |  +1.00  |  -0.83  |  +0.17
  5 ratings of  4 points  |   20/(  5+1) = 3.33  |  (  20+6)/(  5+1) = 4.33  |  +1.00  |  -0.67  |  +0.33
  5 ratings of  3 points  |   15/(  5+1) = 2.50  |  (  15+6)/(  5+1) = 3.50  |  +1.00  |  -0.50  |  +0.50
  5 ratings of  2 points  |   10/(  5+1) = 1.67  |  (  10+6)/(  5+1) = 2.67  |  +1.00  |  -0.33  |  +0.67
  5 ratings of  1 points  |    5/(  5+1) = 0.83  |  (   5+6)/(  5+1) = 1.83  |  +1.00  |  -0.17  |  +0.83
  5 ratings of  0 points  |    0/(  5+1) = 0.00  |  (   0+6)/(  5+1) = 1.00  |  +1.00  |   0.00  |  +1.00
--------------------------+----------------------+---------------------------+---------+---------+---------
 10 ratings of 10 points  |  100/( 10+1) = 9.09  |  ( 100+6)/( 10+1) = 9.64  |  +0.55  |  -0.91  |  -0.36
 10 ratings of  9 points  |   90/( 10+1) = 8.18  |  (  90+6)/( 10+1) = 8.73  |  +0.55  |  -0.82  |  -0.27
 10 ratings of  8 points  |   80/( 10+1) = 7.27  |  (  80+6)/( 10+1) = 7.82  |  +0.55  |  -0.73  |  -0.18
 10 ratings of  7 points  |   70/( 10+1) = 6.36  |  (  70+6)/( 10+1) = 6.91  |  +0.55  |  -0.64  |  -0.09
 10 ratings of  6 points  |   60/( 10+1) = 5.45  |  (  60+6)/( 10+1) = 6.00  |  +0.55  |  -0.55  |   0.00
 10 ratings of  5 points  |   50/( 10+1) = 4.55  |  (  50+6)/( 10+1) = 5.09  |  +0.55  |  -0.45  |  +0.09
 10 ratings of  4 points  |   40/( 10+1) = 3.64  |  (  40+6)/( 10+1) = 4.18  |  +0.55  |  -0.36  |  +0.18
 10 ratings of  3 points  |   30/( 10+1) = 2.73  |  (  30+6)/( 10+1) = 3.27  |  +0.55  |  -0.27  |  +0.27
 10 ratings of  2 points  |   20/( 10+1) = 1.81  |  (  20+6)/( 10+1) = 2.36  |  +0.55  |  -0.19  |  +0.36
 10 ratings of  1 points  |   10/( 10+1) = 0.91  |  (  10+6)/( 10+1) = 1.45  |  +0.55  |  -0.09  |  +0.45
 10 ratings of  0 points  |    0/( 10+1) = 0.00  |  (   0+6)/( 10+1) = 0.55  |  +0.55  |   0.00  |  +0.55
--------------------------+----------------------+---------------------------+---------+---------+---------
100 ratings of 10 points  | 1000/(100+1) = 9.90  |  (1000+6)/(100+1) = 9.96  |  +0.06  |  -0.10  |  -0.04
100 ratings of  9 points  |  900/(100+1) = 8.91  |  ( 900+6)/(100+1) = 8.97  |  +0.06  |  -0.09  |  -0.03
100 ratings of  8 points  |  800/(100+1) = 7.92  |  ( 800+6)/(100+1) = 7.98  |  +0.06  |  -0.08  |  -0.02
100 ratings of  7 points  |  700/(100+1) = 6.93  |  ( 700+6)/(100+1) = 6.99  |  +0.06  |  -0.07  |  -0.01
100 ratings of  6 points  |  600/(100+1) = 5.94  |  ( 600+6)/(100+1) = 6.00  |  +0.06  |  -0.06  |   0.00
100 ratings of  5 points  |  500/(100+1) = 4.95  |  ( 500+6)/(100+1) = 5.01  |  +0.06  |  -0.05  |  +0.01
100 ratings of  4 points  |  400/(100+1) = 3.96  |  ( 400+6)/(100+1) = 4.02  |  +0.06  |  -0.04  |  +0.02
100 ratings of  3 points  |  300/(100+1) = 2.97  |  ( 300+6)/(100+1) = 3.03  |  +0.06  |  -0.03  |  +0.03
100 ratings of  2 points  |  200/(100+1) = 1.98  |  ( 200+6)/(100+1) = 2.04  |  +0.06  |  -0.02  |  +0.04
100 ratings of  1 points  |  100/(100+1) = 0.99  |  ( 100+6)/(100+1) = 1.05  |  +0.06  |  -0.01  |  +0.05
100 ratings of  0 points  |   00/(100+1) = 0.00  |  (  00+6)/(100+1) = 0.06  |  +0.06  |   0.00  |  +0.06

As you can see, the new algorithm will generally increase the total rating and make sure that overall the total rating matches closer to the average, while still providing a noticable weight to entries with more ratings, while entries with fewer ratings are normalized more. It also means that there will probably not be a true 0.00 total rating anymore, which just like a 10.00 total rating would mathematically only be attainable if 1000+ people gave an item a 0.0 respectively 10.0 rating -- and nothing else.

So that's good and exciting, at least for me.


2. Change to the Number of Comments Required per Database Entry
Half a year ago, we set this value to 10, meaning that at least 10 comments must exist for a database entry before a rating without a comment was allowed. This very conservative number was chosen because we did not yet know what the impact of the "open matchguide to everything" functionality would have on the quality and quantity of match ratings. Now, half a year later, I believe we can lower this number back to 5. So that's what I will do.


3. Change to the User Registration Process
Since yesterday it is no longer allowed to use DEAs (Disposable Email Addresses) to register to the site. There are many pros and cons to this decision. But first of all, since not everyone may know, what are disposable email addresses: well, as the name suggests, these are email addresses that are created for a limited time or limited number of emails, before they are automatically destroyed. There are very legitimate uses for DEAs, especially if you want to protect your real email address from websites for privacy or spam protection concerns. I have used them myself in the past and I am using something similar (my own domain -- not cagematch :p -- to use different email addresses per website) to protect my own privacy and emails. I get enough spam via webmaster@cagematch.net, I do not want it on my personal accounts. ^^

However, DEAs were used very, very, very frequently the last couple of months by trolls. By now I guess everyone has seen the one idiot who registers 5-6 accounts per week, rates AEW matches highly and WWE matches lowly. He uses a VPN to mask his IP address and he uses disposable email addresses. I delete his accounts regularly, i.e. every week. For me it's as simple as clicking 10 times to ban 5 accounts and delete their ratings, so whatever he gets out of it, wasting my time is not really part of that. Long story short, not supporting DEAs will not fully get this troll and others like him under control. But it will make it harder for them, waste their time a little bit more.

As for those of you who use DEAs legitimately, I am sorry. We do need to be able to contact you in case of problems with your ratings system behaviour, in which case we are sending emails. Which means we need to be able to contact you, so the email address you use for registering must exist for as long as you are using the ratings system.


4. Zombie User Purge Incoming
Partially because of the troll users, but generally overall we have a lot of dead user accounts in our system. I am not talking about users who once participated in the ratings system and then left, these are a real part of our history and an important part of our ability to show ratings histores for database entries. I am talking about user accounts that have never participated in the ratings system and often never logged in after registration. Of the 54155 registered users, only a fifth has provided ratings. The rest are bots, trolls and others with such illustrious names like "zpqmxwzob" and I hate that they clutter up the database. So I am going to delete them, as I also do not really care that much about the "OMG 50000 USER ACCOUNTS!", I rather care about the "OMG 10000 REAL PEOPLE!" accomplishment. I do not make money off the site and I have not compared our numbers to other sites for a very long time, so I am just going to go ahead and delete the zombie accounts, so that even some legitimate user names like "AJStyles" are actually available again for users who actually participate in the ratings system.

I will make sure that no user accounts are deleted that have:
- at least one rating or comment
- at least one notebook entry
- registered within the last 30 days with a non-disposable email address (as these users might become active now during the holidays)


That's it for now. Enjoy your day. :)
Send any criticism, love or hate about these changes my way, if you want, but my mind is pretty much made up.

Cheers,
Philip
(Matt Macks)