Some thoughts on ranking systems

For topics that last throughout the whole season
User avatar
maglor
~Fukou da~
~Fukou da~
Posts: 8614
Joined: Thu Feb 26, 2009 8:57 pm
Badges:
ImageImage
Worships: Abriel Nei Debrusc Borl Paryun Lafiel
Melon Pan: 75
2018 Female Favorite: Chtholly Nota Seniorious
2018 Male Favorite: Yang Wenli
2017 Female Favorite: Tomori Nao
2017 Male Favorite: Yang Wenli
Wish: More people being open to alternatives and compromises.

Re: Some thoughts on ranking systems

Post by maglor » Tue Jan 20, 2015 7:31 am

Shmion and I worked on another score system which may be a good indicator of people's expectation of the character. After some thought I decided to call it ISML Character's Expectation Value, or The Value for the short.

Theoretical background

The score system is inspired by

(1) Pythagorean Expectations => http://en.wikipedia.org/wiki/Pythagorean_expectation" onclick="window.open(this.href);return false;
(2) Random Walk => http://rankings.amath.unc.edu/old/monkeys.htm" onclick="window.open(this.href);return false;

Calculation

(1) The base assumption is that there exist an Pythagorean Expectation formula that can connect votes for and votes against to the Win% of a character. Furthermore, this win% may be close to what people expect from a character in an ISML Match. After many trial and error, it was decided that the following formula is the simplest formula that adequately fits ISML results

Expected Win% = 1 / [ 1 + ( VA/VF) ^ 5 ]

The Root Mean Square Error = sqrt [ sum of all of (Expected Win% - Actual Win% ) ^2 ] = 0.066326

The RMSE value corresponds to average error of being off by about two wins in typical ISML season, but any formula that reduced error involved much more factors, which would have made this part very cumbersome. The form above was chosen for its simplicity, and also because the number 5 played important role in the next part.

(2) The next assumption is how much of an expectation that a character carries into the match is at the stake for a match. It is like how much of the "Pot" the character has will be "Bet" into the match pool. Stronger characters are likely to carry a bigger "Pot" thus have more expectation at stake for a match. How much of the "Pot" was "bet" on each match was of great importance, as "betting" too little would make the "Expectation Value" change too slowly, while "betting" too much would make it very volatile and not mirror the reality.

Shmion and I tested the "bet" amount between 0.01 = 1% of the whole pot to 0.5=50% of the whole pot on 2014 Nova and Stella Data. The calculation becomes as follow
  • i) All characters start with 100 points as their Pot
    ii) In each match character bets N% of the Pot into the Match Pot. We explored N%=0.01=1% to N%=0.5=50%
    iii) After the match between Character A and B, they will divide up the match pot by the following equation

    Amount of Match Pot earned by character A = Match Pot Value / [ 1 + ( Vote for character B / Vote for character A ) ^5 ]

    iv) The character's pot value will be readjusted and they will go to the next match
To give more concrete example, here is how the first few calculation would look like

i) Character A and B would have pot value of 100 at the start of the season
ii) In the first match of the season they are against each other. Each put up Pot * N% = 100*N% into the match pot. The Match Pot Value is (100 ( = Pot_A ) +100( = Pot_B) )*N%
iii) In the match Character A got Vf_A votes and Character B got Vf_B votes
iv) The Character A's Pot after 1st match would be New Pot value = 100 ( = initial pot ) - 100*N% ( Amount placed into the match pot ) + (100 ( = Pot_A ) +100( = Pot_B) )*N% / [ 1 + ( Vf_B / Vf_A ) ^5 ]

Shmion and I ranked the characters by their expectation value at the end of the season and compared it to our traditional point ranking. We wanted to find a case where average rank difference between expectation value and traditional point ranking was bigger than 1, but less than 5. We also didn't want more than two cases where rank difference would be more than 10. Another constraint I looked at was that maximum amount of "Pot" change that occurred from a match be near 50, where this 50 was chosen as value that is about half the initial pot value.

What we found was that when N% is greater than 0.2, we get the average rank difference to be greater than 1, but things somewhat stabilize after that, thus we couldn't exceed average rank difference of 2 even for N% = 0.5 in case of Nova ( 2.333 for Stella which is still near 2 ) . However, maximum amount of Pot change was as great as 174 for Nova and 232 for Stella, which means there was potential for it to be overly volatile. After much simulation, N% = 1/5 = 0.2 was chosen to be optimal. The value of 0.2 may not be mere coincidence as the exponent in our Pythagorean Expectation formula also was 5.

You can see the actual numbers at https://docs.google.com/spreadsheets/d/ ... 1236600324" onclick="window.open(this.href);return false; which was made by Shmion .

3) discussions

i) Illya at the end of 2014 Stella showed significant rise in Expectation value ranks. I believe the increase to be close to apparent level of interest people gave Illya due to her Topaz wins.
ii) Depending on how much characters placed into match pot, it is possible for a character to gain expectation value even when she lost. This is akin to a real world sports case where a team that used to struggle starts doing suddenly well, and in an important match, nearly beat an opponent that was considered to be one of the best team in the league. Many people would start paying more attention to the team that exceeded expectations.
iii) Unlike 1%-35% point stealing scheme, nor our traditional point system, every vote really matters, WIN OR LOSS . While this is great for voter participation, it also makes this "Value" system be vulnerable to manipulations by small yet consistent faction voting.
iv) The value rank should move faster than traditional point rank when a character receives a late season push. This can be very useful for ********* .
v) The Value and Value Rank can replace the SVAO and SWVO column in the stat table. VP and SVDO column should be replaced by the 1%-35% Score column and the Score Rank.
vi) It should be noted that the Value tracks expectation, not the actual strength level of a character. There already are many Actual Strength Level calculation systems that beats this Value system in predicting the results.

----- Conclusion ----

I believe we have found a nice formula that somewhat matches amount of hype a character may have. This hype amount can be used in various ways. It is recommended that the columns for the Expectation Value and the Value rank replace two of the columns in current stat table so its evolution can be watched and analyzed by the public.
Image
User avatar
avery-kun
Moon princess
Moon princess
Posts: 3128
Joined: Thu Oct 04, 2012 4:25 am
Badges:
ImageImageImage
Worships: Berserker!
Melon Pan: 135
2017 Female Favorite: Illyasviel Von Einzbern
2017 Male Favorite: Gilgamesh
Wish: Illya 2017!

Re: Some thoughts on ranking systems

Post by avery-kun » Wed Feb 04, 2015 8:35 pm

I think I'm gonna steal this as a scoring system for a game.
Image
User avatar
Chocola
Hikarin's Kitty
Hikarin's Kitty
Posts: 7449
Joined: Tue Feb 17, 2009 3:31 am
Worships: Kousaka Reina
Melon Pan: 50
2017 Female Favorite: Karasuma Chitose
2017 Male Favorite: Killua Zoldyck
Wish: Hikachu to not hate me

Re: Some thoughts on ranking systems

Post by Chocola » Thu Feb 05, 2015 2:45 am

avery-kun wrote:I think I'm gonna steal this as a scoring system for a game.
Yay, someone's gonna steal a great system I proposed that is mathematically sound but people reject because they don't understand it. Good luck with whatever it is you're doing. o.o
ImageImageImage
ImageImageImage
MAL Ratings ISML Chat, Saimoe Blog, Saimoe Wiki
A tall, towering wall looms in front of me. Beyond that is something that I could never to see on my own.
And that is...the view from the top.
User avatar
Desufire
Shinigami
Shinigami
Posts: 2633
Joined: Tue Dec 18, 2012 1:14 pm
Badges:
Image
Worships: Orihime Inoue
Melon Pan: 60
Wish: For sugar to not be unhealthy, and campaigning to be easier
Location: Vancouver

Re: Some thoughts on ranking systems

Post by Desufire » Thu Feb 05, 2015 3:41 am

This is YOUR GUYS' website. Input ANY rules you want. If people leave because of it, screw them, you've explained how fair it is and they're just having a tantrum because you took away one of their toys and gave them a better one.
Image
User avatar
avery-kun
Moon princess
Moon princess
Posts: 3128
Joined: Thu Oct 04, 2012 4:25 am
Badges:
ImageImageImage
Worships: Berserker!
Melon Pan: 135
2017 Female Favorite: Illyasviel Von Einzbern
2017 Male Favorite: Gilgamesh
Wish: Illya 2017!

Re: Some thoughts on ranking systems

Post by avery-kun » Thu Feb 05, 2015 10:24 pm

Kholdy wrote:
avery-kun wrote:I think I'm gonna steal this as a scoring system for a game.
Yay, someone's gonna steal a great system I proposed that is mathematically sound but people reject because they don't understand it. Good luck with whatever it is you're doing. o.o
Well its ISML related, so hopefully I'm not too lazy/stupid to finalize a scoring system that does what I want before Prelims are over. Or learn how to make (decent) posters...
Image
User avatar
Homura
[Apprentice Magician]
[Apprentice Magician]
Posts: 3012
Joined: Wed Sep 25, 2013 3:25 am
Badges:
Melon Pan: 55
2018 Female Favorite: Makise Kurisu
2018 Male Favorite: Yukihira Sōma
2017 Female Favorite: Katō Megumi
2017 Male Favorite: Yukihira Sōma
Location: China

Re: Some thoughts on ranking systems

Post by Homura » Thu Feb 05, 2015 11:13 pm

Kholdy wrote:
avery-kun wrote:I think I'm gonna steal this as a scoring system for a game.
Yay, someone's gonna steal a great system I proposed that is mathematically sound but people reject because they don't understand it. Good luck with whatever it is you're doing. o.o
Will one day Kholdy's name be seen in textbooks as one section title Kholdy's System? (then millions of students will start blaming him because of hard exam questions XP )
User avatar
Shmion84
Moon princess
Moon princess
Posts: 3383
Joined: Mon Apr 02, 2012 6:33 pm
Badges:
ImageImage
Melon Pan: 95
2017 Female Favorite: Konjiki No Yami
2017 Male Favorite: Kyon
Wish: ISML still exists next year
Location: Central Europe

Re: Some thoughts on ranking systems

Post by Shmion84 » Mon Feb 09, 2015 9:51 pm

Small test of new necklace formula (2015 Charter IX.) using 2014 data and under the condition the overall necklace percentage don't change.

Aquamarine Necklace:
Misaka Mikoto - 61.01 pts - no change
Amethyst Necklace:
Tachibana Kanade - 60.30 pts - no change
Ruby Necklace:
Kuroyukihime - 47.72 pts - no change
Emerald Necklace:
Yūki Asuna - 52.44 pts - no change
Topaz Necklace:
Shana - 52.40 pts - original winner: Shiina Mashiro with 52.22 pts
Image
User avatar
Homura
[Apprentice Magician]
[Apprentice Magician]
Posts: 3012
Joined: Wed Sep 25, 2013 3:25 am
Badges:
Melon Pan: 55
2018 Female Favorite: Makise Kurisu
2018 Male Favorite: Yukihira Sōma
2017 Female Favorite: Katō Megumi
2017 Male Favorite: Yukihira Sōma
Location: China

Re: Some thoughts on ranking systems

Post by Homura » Mon Feb 09, 2015 11:20 pm

That hurts...
User avatar
Chocola
Hikarin's Kitty
Hikarin's Kitty
Posts: 7449
Joined: Tue Feb 17, 2009 3:31 am
Worships: Kousaka Reina
Melon Pan: 50
2017 Female Favorite: Karasuma Chitose
2017 Male Favorite: Killua Zoldyck
Wish: Hikachu to not hate me

Re: Some thoughts on ranking systems

Post by Chocola » Tue Feb 10, 2015 12:05 am

Remember the intangible factors everyone. For example, you might not have voted for someone because under the old SDO format, that character looks unwinnable and you would not want to "waste" your votes. If the old necklace matches were instead displayed in PSAO, then suddenly there are more "winnable" candidates and the results could be much different.

Hypothetically.

So yes, Azusa would've won. I believe!
ImageImageImage
ImageImageImage
MAL Ratings ISML Chat, Saimoe Blog, Saimoe Wiki
A tall, towering wall looms in front of me. Beyond that is something that I could never to see on my own.
And that is...the view from the top.
User avatar
avery-kun
Moon princess
Moon princess
Posts: 3128
Joined: Thu Oct 04, 2012 4:25 am
Badges:
ImageImageImage
Worships: Berserker!
Melon Pan: 135
2017 Female Favorite: Illyasviel Von Einzbern
2017 Male Favorite: Gilgamesh
Wish: Illya 2017!

Re: Some thoughts on ranking systems

Post by avery-kun » Tue Feb 10, 2015 12:12 am

I wonder how having two votes will change things in the necklace showdown. Now I can vote for both Mikoto AND Kanade. At least I would, if they were still in the tournament!

This could mean that people like Kotori or Yoshino could have disturbed the percentages a lot more, and maybe have won a necklace. Or it could have made things worst for them.
Image
User avatar
Homura
[Apprentice Magician]
[Apprentice Magician]
Posts: 3012
Joined: Wed Sep 25, 2013 3:25 am
Badges:
Melon Pan: 55
2018 Female Favorite: Makise Kurisu
2018 Male Favorite: Yukihira Sōma
2017 Female Favorite: Katō Megumi
2017 Male Favorite: Yukihira Sōma
Location: China

Re: Some thoughts on ranking systems

Post by Homura » Tue Feb 10, 2015 12:27 am

Kholdy wrote:Remember the intangible factors everyone. For example, you might not have voted for someone because under the old SDO format, that character looks unwinnable and you would not want to "waste" your votes. If the old necklace matches were instead displayed in PSAO, then suddenly there are more "winnable" candidates and the results could be much different.

Hypothetically.

So yes, Azusa would've won. I believe!
Azusa's no necklace might be the biggest "failure" of ISML so far (but no one can exactly tell what the "failure" is). Do you agree?
User avatar
smartboyhw
[Future Overlord]
[Future Overlord]
Posts: 3260
Joined: Fri May 16, 2014 2:18 pm
Badges:
ImageImage
Worships: Isshiki Iroha
Melon Pan: 0
2018 Female Favorite: Isshiki Iroha
2018 Male Favorite: Killua Zoldyck
2017 Female Favorite: Isshiki Iroha
2017 Male Favorite: Killua Zoldyck
Wish: Shichimiya Satone, Aoyama Nanami and Nakano Azusa to win an ISML prize, Inaba Himeko to win BGC and live in Sakura-sō
Location: Hong Kong, China

Re: Some thoughts on ranking systems

Post by smartboyhw » Tue Feb 10, 2015 1:04 am

Kholdy wrote:Remember the intangible factors everyone. For example, you might not have voted for someone because under the old SDO format, that character looks unwinnable and you would not want to "waste" your votes. If the old necklace matches were instead displayed in PSAO, then suddenly there are more "winnable" candidates and the results could be much different.

Hypothetically.

So yes, Azusa would've won. I believe!
ImageImage

Let me teach you something: We humans... Even for 10 minutes... We cannot even wait that long! - Nakamura Yuri, Angel Beats!
User avatar
matchbaby
Necromancer
Necromancer
Posts: 887
Joined: Sat Jun 28, 2014 9:51 am
Badges:
ImageImage
Worships: Isla
Melon Pan: 65
Wish: Isla!!! All hail for Isla!!!
Shiro and Sora!! As least win something!!
Same as Illya~!
Location: Hong Kong, China

Re: Some thoughts on ranking systems

Post by matchbaby » Tue Feb 10, 2015 10:54 am

Homura wrote:
Kholdy wrote:Remember the intangible factors everyone. For example, you might not have voted for someone because under the old SDO format, that character looks unwinnable and you would not want to "waste" your votes. If the old necklace matches were instead displayed in PSAO, then suddenly there are more "winnable" candidates and the results could be much different.

Hypothetically.

So yes, Azusa would've won. I believe!
Azusa's no necklace might be the biggest "failure" of ISML so far (but no one can exactly tell what the "failure" is). Do you agree?
I can't disagree -_- :arrowed:
User avatar
avery-kun
Moon princess
Moon princess
Posts: 3128
Joined: Thu Oct 04, 2012 4:25 am
Badges:
ImageImageImage
Worships: Berserker!
Melon Pan: 135
2017 Female Favorite: Illyasviel Von Einzbern
2017 Male Favorite: Gilgamesh
Wish: Illya 2017!

Re: Some thoughts on ranking systems

Post by avery-kun » Tue Feb 10, 2015 2:41 pm

Homura wrote:
Kholdy wrote:Remember the intangible factors everyone. For example, you might not have voted for someone because under the old SDO format, that character looks unwinnable and you would not want to "waste" your votes. If the old necklace matches were instead displayed in PSAO, then suddenly there are more "winnable" candidates and the results could be much different.

Hypothetically.

So yes, Azusa would've won. I believe!
Azusa's no necklace might be the biggest "failure" of ISML so far (but no one can exactly tell what the "failure" is). Do you agree?
I don't understand the "failure".
Image
User avatar
maglor
~Fukou da~
~Fukou da~
Posts: 8614
Joined: Thu Feb 26, 2009 8:57 pm
Badges:
ImageImage
Worships: Abriel Nei Debrusc Borl Paryun Lafiel
Melon Pan: 75
2018 Female Favorite: Chtholly Nota Seniorious
2018 Male Favorite: Yang Wenli
2017 Female Favorite: Tomori Nao
2017 Male Favorite: Yang Wenli
Wish: More people being open to alternatives and compromises.

Re: Some thoughts on ranking systems

Post by maglor » Thu Mar 26, 2015 8:55 pm

maglor wrote:
saving spaceShow
Shmion and I worked on another score system which may be a good indicator of people's expectation of the character. After some thought I decided to call it ISML Character's Expectation Value, or The Value for the short.

Theoretical background

The score system is inspired by

(1) Pythagorean Expectations => http://en.wikipedia.org/wiki/Pythagorean_expectation" onclick="window.open(this.href);return false;
(2) Random Walk => http://rankings.amath.unc.edu/old/monkeys.htm" onclick="window.open(this.href);return false;

Calculation

(1) The base assumption is that there exist an Pythagorean Expectation formula that can connect votes for and votes against to the Win% of a character. Furthermore, this win% may be close to what people expect from a character in an ISML Match. After many trial and error, it was decided that the following formula is the simplest formula that adequately fits ISML results

Expected Win% = 1 / [ 1 + ( VA/VF) ^ 5 ]

The Root Mean Square Error = sqrt [ sum of all of (Expected Win% - Actual Win% ) ^2 ] = 0.066326

The RMSE value corresponds to average error of being off by about two wins in typical ISML season, but any formula that reduced error involved much more factors, which would have made this part very cumbersome. The form above was chosen for its simplicity, and also because the number 5 played important role in the next part.

(2) The next assumption is how much of an expectation that a character carries into the match is at the stake for a match. It is like how much of the "Pot" the character has will be "Bet" into the match pool. Stronger characters are likely to carry a bigger "Pot" thus have more expectation at stake for a match. How much of the "Pot" was "bet" on each match was of great importance, as "betting" too little would make the "Expectation Value" change too slowly, while "betting" too much would make it very volatile and not mirror the reality.

Shmion and I tested the "bet" amount between 0.01 = 1% of the whole pot to 0.5=50% of the whole pot on 2014 Nova and Stella Data. The calculation becomes as follow
  • i) All characters start with 100 points as their Pot
    ii) In each match character bets N% of the Pot into the Match Pot. We explored N%=0.01=1% to N%=0.5=50%
    iii) After the match between Character A and B, they will divide up the match pot by the following equation

    Amount of Match Pot earned by character A = Match Pot Value / [ 1 + ( Vote for character B / Vote for character A ) ^5 ]

    iv) The character's pot value will be readjusted and they will go to the next match
To give more concrete example, here is how the first few calculation would look like

i) Character A and B would have pot value of 100 at the start of the season
ii) In the first match of the season they are against each other. Each put up Pot * N% = 100*N% into the match pot. The Match Pot Value is (100 ( = Pot_A ) +100( = Pot_B) )*N%
iii) In the match Character A got Vf_A votes and Character B got Vf_B votes
iv) The Character A's Pot after 1st match would be New Pot value = 100 ( = initial pot ) - 100*N% ( Amount placed into the match pot ) + (100 ( = Pot_A ) +100( = Pot_B) )*N% / [ 1 + ( Vf_B / Vf_A ) ^5 ]

Shmion and I ranked the characters by their expectation value at the end of the season and compared it to our traditional point ranking. We wanted to find a case where average rank difference between expectation value and traditional point ranking was bigger than 1, but less than 5. We also didn't want more than two cases where rank difference would be more than 10. Another constraint I looked at was that maximum amount of "Pot" change that occurred from a match be near 50, where this 50 was chosen as value that is about half the initial pot value.

What we found was that when N% is greater than 0.2, we get the average rank difference to be greater than 1, but things somewhat stabilize after that, thus we couldn't exceed average rank difference of 2 even for N% = 0.5 in case of Nova ( 2.333 for Stella which is still near 2 ) . However, maximum amount of Pot change was as great as 174 for Nova and 232 for Stella, which means there was potential for it to be overly volatile. After much simulation, N% = 1/5 = 0.2 was chosen to be optimal. The value of 0.2 may not be mere coincidence as the exponent in our Pythagorean Expectation formula also was 5.

You can see the actual numbers at https://docs.google.com/spreadsheets/d/ ... 1236600324" onclick="window.open(this.href);return false; which was made by Shmion .

3) discussions

i) Illya at the end of 2014 Stella showed significant rise in Expectation value ranks. I believe the increase to be close to apparent level of interest people gave Illya due to her Topaz wins.
ii) Depending on how much characters placed into match pot, it is possible for a character to gain expectation value even when she lost. This is akin to a real world sports case where a team that used to struggle starts doing suddenly well, and in an important match, nearly beat an opponent that was considered to be one of the best team in the league. Many people would start paying more attention to the team that exceeded expectations.
iii) Unlike 1%-35% point stealing scheme, nor our traditional point system, every vote really matters, WIN OR LOSS . While this is great for voter participation, it also makes this "Value" system be vulnerable to manipulations by small yet consistent faction voting.
iv) The value rank should move faster than traditional point rank when a character receives a late season push. This can be very useful for ********* .
v) The Value and Value Rank can replace the SVAO and SWVO column in the stat table. VP and SVDO column should be replaced by the 1%-35% Score column and the Score Rank.
vi) It should be noted that the Value tracks expectation, not the actual strength level of a character. There already are many Actual Strength Level calculation systems that beats this Value system in predicting the results.

----- Conclusion ----

I believe we have found a nice formula that somewhat matches amount of hype a character may have. This hype amount can be used in various ways. It is recommended that the columns for the Expectation Value and the Value rank replace two of the columns in current stat table so its evolution can be watched and analyzed by the public.
We will need some volunteers to keep track of the relevant values in the above post for the entire regular season, each individual necklace period, postseason overall, and each postseason phases. One thing to remember is that at the start of each new period, all players have 100.00 "gold units" in their pot. If you help out, although lolipops aren't worth much, I will beg Crisu to give 100 lolipops for each table you generate for each division. ( do all 3 division and you get 300 lolipops )
Image
User avatar
Kordosa
Soul gem
Soul gem
Posts: 4062
Joined: Mon Apr 19, 2010 3:35 am
Worships: Moriya Suwako's hat
Melon Pan: 50
2018 Female Favorite: Yamada Elf
2018 Male Favorite: Willem Kmetsch
2017 Female Favorite: Darkness
2017 Male Favorite: Roy Mustang
Wish: A third season of Spice and Wolf.
Location: Mississippi, USA
Contact:

Re: Some thoughts on ranking systems

Post by Kordosa » Thu Mar 26, 2015 11:13 pm

Thread: TL;DR
maglor's above post: TL;DR. Also, TMM;DR (too much math; didn't read).
maglor wrote:I believe we have found a nice formula that somewhat matches amount of hype a character may have. This hype amount can be used in various ways.
I'm going to mention something that has been mentioned before on several occasions. Hype is overrated. The ratio of "good" Nova series represented in the regular season vs. "bad" Nova series represented in the regular season seems to be tipped toward the more mediocre series that have just aired/are currently airing. Whether that is actually objectively true or not is an entirely different thing. But I have to assume there are many voters out there who at least have the perception that Nova is simply where the most recent, most generic series get most of the regular season slots.

Since I have not read through the entirety of this thread, nor read any of maglor's mathematical jargon, I have no real opinion about whatever new formula will be used, other than I hope it's pretty much straightfoward and allows for a fair system. I'm not a fan of overly complicated or convuluted systems; I believe in functional simplicity; however, I do recognize that there is a trade-off between simplicity and providing fairness in the system.

Anyway, I was just merely pointing something out that is likely a sore point among several voters, so mentioning "hype" being taken into account for whatever new formula that has been devised could be taken several different ways by the voters, if not explained a bit more clearly. If it already has been explained earlier in the thread, I apologize.
QuotesShow
Cirno wrote:*sinister laugh* Winning by only 47 votes is all part of my master plan. Now everyone will think I'm weak when, in fact, I'm the strongest. And then, when they least expect it, I'll strike back and take over the entire ISML. It's foolproof. Hahahaha, I'm such a genius!
Crisu wrote:And, of course, never merge an anti-cookie with a normal cookie. Serious consequence will occur.
shiraoky wrote:I'm always squeeing lol.
Metaler wrote:Seriously, if you're gonna do something badly, then don't bother doing it. It's like when you take a dump: you don't show it to other people specifically because it's shit!
Kordosa wrote:Protip: If a male high school student character is voiced by a female seiyuu, there is a 100% probability that that character will be forced to crossdress at some point.
User avatar
matchbaby
Necromancer
Necromancer
Posts: 887
Joined: Sat Jun 28, 2014 9:51 am
Badges:
ImageImage
Worships: Isla
Melon Pan: 65
Wish: Isla!!! All hail for Isla!!!
Shiro and Sora!! As least win something!!
Same as Illya~!
Location: Hong Kong, China

Re: Some thoughts on ranking systems

Post by matchbaby » Sat Mar 28, 2015 4:23 am

I look into the useless SWVO data and I notice a problem
"SWVO lift for each match has a strong positive correlation with SVAO(Strength of Opponent), which some of the R-value excess 0.8 or even 0.9, mean absolute R-value is 0.5"

SWVO, a indicator for character's strength, has no meaning on rank, necklace or others, however, I don't think it is meaningless as my old model uses SWVO to predict ^^;
What I suppose is change some minor things inside the SWVO equation.

currently, SWVO = sum of ( VP of match * (that sum of opponent's VP in the previous n matches))
However, the SWVO lift for each match varies a lot (this is not a problem) and correlate with SVAO (This I think is a problem)
Moreover, for predicting results (by trial and error), the best equation is SWVO of A^1.75/(SWVO of A^1.75+ SWVO of B^1.75), but what does that number 1.75 means? It has no physical meaning (I think) , just a constant.
If I use a new equation
new SWVO = sum of ( New VP of match * (that sum of opponent's New VP in the previous n matches)) where New VP = old VP* VF% of that match

The correlation of new SVAO and new SWVO becomes very small, mean absolute R-value is 0.3
Beside, for predicting result, just use newSWVO of A / (newSWVO of A + newSWVO of B), no more constant is needed. (And this shows a more accurate prediction than the old one)

I don't know the reason behind why the number fits the prediction that good with changing this minor things ( I suddenly get this idea when I am dreaming -_- ), can someone explain to me...
Well, maybe you can try yourself so you can know more about this new SWVO system.
User avatar
Just
[Kiwi's Melon]
[Kiwi's Melon]
Posts: 3664
Joined: Wed Apr 10, 2013 11:40 am
Badges:
ImageImageImageImage
Melon Pan: 0
2017 Female Favorite: Shiodome Miuna
2017 Male Favorite: Hikigaya Hachiman
Wish: May children never suffer.
Location: Hong Kong

Re: Some thoughts on ranking systems

Post by Just » Sat Mar 28, 2015 1:37 pm

Read maglor's post, sounds interesting.

So the job is to keep track of the Pot values for each character with each match, right? The RMSE won't be calculated until the end of each specified period (i.e. RS, Necklace Period etc.)
(I guess it's something similar to a character 'strength' estimation model that some of us used for Fantasy last year, though probably with more scientific and mathematical basis.)

I have confidence that matchbaby would be interested in and have the time to do this.
Image
SpoilerShow
Image
Image Image Image Image Image Image
User avatar
maglor
~Fukou da~
~Fukou da~
Posts: 8614
Joined: Thu Feb 26, 2009 8:57 pm
Badges:
ImageImage
Worships: Abriel Nei Debrusc Borl Paryun Lafiel
Melon Pan: 75
2018 Female Favorite: Chtholly Nota Seniorious
2018 Male Favorite: Yang Wenli
2017 Female Favorite: Tomori Nao
2017 Male Favorite: Yang Wenli
Wish: More people being open to alternatives and compromises.

Re: Some thoughts on ranking systems

Post by maglor » Sat Mar 28, 2015 8:16 pm

Just wrote:Read maglor's post, sounds interesting.

So the job is to keep track of the Pot values for each character with each match, right? The RMSE won't be calculated until the end of each specified period (i.e. RS, Necklace Period etc.)
(I guess it's something similar to a character 'strength' estimation model that some of us used for Fantasy last year, though probably with more scientific and mathematical basis.)

I have confidence that matchbaby would be interested in and have the time to do this.
Pot value for each character after each character is what's needed. RMSE isn't of concern as we already know this method is inferior to some other methods when it comes to character strength prediction.
Image
User avatar
Shmion84
Moon princess
Moon princess
Posts: 3383
Joined: Mon Apr 02, 2012 6:33 pm
Badges:
ImageImage
Melon Pan: 95
2017 Female Favorite: Konjiki No Yami
2017 Male Favorite: Kyon
Wish: ISML still exists next year
Location: Central Europe

Re: Some thoughts on ranking systems

Post by Shmion84 » Thu Apr 02, 2015 2:45 pm

matchbaby wrote:I look into the useless SWVO data and I notice a problem
"SWVO lift for each match has a strong positive correlation with SVAO(Strength of Opponent), which some of the R-value excess 0.8 or even 0.9, mean absolute R-value is 0.5"

SWVO, a indicator for character's strength, has no meaning on rank, necklace or others, however, I don't think it is meaningless as my old model uses SWVO to predict ^^;
What I suppose is change some minor things inside the SWVO equation.

currently, SWVO = sum of ( VP of match * (that sum of opponent's VP in the previous n matches))
However, the SWVO lift for each match varies a lot (this is not a problem) and correlate with SVAO (This I think is a problem)
Moreover, for predicting results (by trial and error), the best equation is SWVO of A^1.75/(SWVO of A^1.75+ SWVO of B^1.75), but what does that number 1.75 means? It has no physical meaning (I think) , just a constant.
If I use a new equation
new SWVO = sum of ( New VP of match * (that sum of opponent's New VP in the previous n matches)) where New VP = old VP* VF% of that match

The correlation of new SVAO and new SWVO becomes very small, mean absolute R-value is 0.3
Beside, for predicting result, just use newSWVO of A / (newSWVO of A + newSWVO of B), no more constant is needed. (And this shows a more accurate prediction than the old one)

I don't know the reason behind why the number fits the prediction that good with changing this minor things ( I suddenly get this idea when I am dreaming -_- ), can someone explain to me...
Well, maybe you can try yourself so you can know more about this new SWVO system.
I also cannot explain why this new formula is working fine (I didn't test it until now...), but I can offer one hint:
(1)

Code: Select all

new VP = old VP * VF%
(2)

Code: Select all

old VP = VF / ave VF
(3)

Code: Select all

VF% = VF / (VF + VA)
If we assume, that the total votes don't vary much among the arenas, we get:
(4)

Code: Select all

2 * ave VF = VF + VA
We substitute this term in (3):
(5)

Code: Select all

VF% = VF / (2 * ave VF) = old VP / 2
We finally get:
(6)

Code: Select all

new VP = old VP * old VP / 2 = (old VP)^2 / 2
I would appreciate, if you could keep track and post new SWVO values in Regular Season.
Image
Post Reply