Virtual prediction contest (Ahwaz Airport game #4)
in [World Football]
|
From: El Kot on 24 Mar 2010 21:10 milivella wrote: > El Kot: > >> We can use the interim EPL table to judge how good FM's simulation >> is. Frankly, I'm not impressed. You can compare the simulated table with >> the real one at Kicker: >> >> http://www.kicker.de/news/fussball/intligen/england/barclays-premier-... >> >> One would expect the Fulham results to be different from Fulmark's, but >> the rest of the teams should have performed in simulation more or less >> similar to reality. > > You are highlighting an interesting point. Leaving aside some banal > considerations (e.g. a. the comparison will be most interesting at the > end of the season, after 380 matches; b. you can have a bad > simulation, or no simulation at all, and still produce a table that is > similar to reality), I have some questions: Hehe, your point b) is funny. One judges a simulation by its outcome; if a simulation produces a result that is the same as what it simulates, then it is, by definition, good. :) Here is what I think about the points you raise: - the real match outcomes and table are a sampling of the teams' real strengths. They are, of course, subject to randomness; individual matches can be influenced by chance. However, repeated sampling will converge to the true value, because I don't think the influence of chance is /that/ big that it dominates; the variance is not that big. Remember that a sequence is a product of chances; you can't expect the unexpected to happen too often in a row. - the simulation has the advantage that it can be repeated and its statistics checked out. If and when you have time, please make several runs, and check the following things: 1. How similar to each other the simulated tables are at the end of the season. This will tell us how consistently the FM engine applies the parameters of the players and teams. If the tables turn out to be more or less similar (with each team moving by no more than, let's say, 2 to 3 positions around the table), we can conclude that it is a consistent engine. 2. Then we can check how true to life it is, by comparing the actual position from real life to the simulated ones. This will tell us how realistic the parameters and methods it uses are. 3. And of course we can see how it is influenced by randomness, by looking at the outcomes of one match played several times. This is allowed to have a bigger variance, but still within limits. One wouldn't expect ManU to beat Portsmouth less than 8 times out of 10. - what you say below, is backwards. We can't really calculate the statistics of the real teams; there's too few samples. We can, however, calculate the statistics of the simulation, and then see if the real results fall within the predicted confidence intervals. And that will tell us how true to life the simulation really is - if the predicted points per game for ManU is x points, and the actual one is y, we can check to see within which confidence interval of the simulation that value falls, and deduce the quality of the simulation. I'm not a statistician, so I will abstain from claiming that, if the actual results of the teams fall within the 80% confidence intervals, it means that the simulation's accuracy is 80%. The relationship is surely more complicated; if Daniele or someone else (MH?) knows more, please share. > 1. Is it correct to consider the real EPL table like a sample of > teams' real value (rather than a perfect representation of it)? After > all, one could say that, if Blackburn and Chelsea play 10 times, > Chelsea would win 6 matches (random example), so their draw is like 1 > of the other 4 matches, i.e. it's a sample of the 10 hypothetical > matches between the two teams. > > 2. If the answer to (1) is "yes": we can compute a confidence > interval, can't we? I.e. we can say "with a confidence level of 95%, > Manchester United's real value measured in points per game is 2.23 (= > 69 points / 31 matches) plus or minus x". > > 3. Things are of course easier in FM, because there we can simulate > the same season a lot of times (I just have to remember to check the > game after a couple of hours, save the standings and re-start the > season). After having ran a lot of simulations and having averaged the > results, we should be able to check if the simulated results are > inside or outside the confidence interval mentioned at (2), so > verifying/falsifying (in the limits of what it possible) FM's claims > of realism. Am I wrong? -- No, no, you can't e-mail me with the nono.
From: milivella on 25 Mar 2010 10:09 milivella: > 1st update: September 16: Fulham is 18th. 2nd update: October 16: Fulham is still 18th. Data-filled screenshots: Played matches: http://i.imgur.com/FN4Y7.jpg Competitions: - Premier League: http://i.imgur.com/v6F3j.jpg - Europa League: http://i.imgur.com/lNCbJ.jpg - [out of League Cup] Players' stats: - Goalkeepers: http://i.imgur.com/v8YV9.jpg - Defenders: http://i.imgur.com/8jvvU.jpg - Midfielders: http://i.imgur.com/3xgwt.jpg - Attackers: http://i.imgur.com/u70rm.jpg -- Cheers milivella
From: Jesus Petry on 25 Mar 2010 10:37 On Mar 25, 11:09 am, milivella <milive...(a)gmail.com> wrote: > milivella: > > > 1st update: September 16: Fulham is 18th. > > 2nd update: October 16: Fulham is still 18th. > > Data-filled screenshots: > Played matches:http://i.imgur.com/FN4Y7.jpg > Competitions: > - Premier League:http://i.imgur.com/v6F3j.jpg > - Europa League:http://i.imgur.com/lNCbJ.jpg But I see they have games in hand and are doing well in Europe! Tchau! Jesus Petry
From: milivella on 25 Mar 2010 10:39 El Kot: Thanks for the reply. > Hehe, your point b) is funny. One judges a simulation by its > outcome; if a simulation produces a result that is the same as what it > simulates, then it is, by definition, good. :) OK. Question: if, at the beginning of the season, you ask me tiopredict the final table and I just copy the previous season's final table (replacing relegated teams), is it a simulation? (more questions to come if you answer "yes"...) > - the real match outcomes and table are a sampling of the teams' real > strengths. They are, of course, subject to randomness; individual > matches can be influenced by chance. However, repeated sampling will > converge to the true value, because I don't think the influence of > chance is /that/ big that it dominates; the variance is not that big. > Remember that a sequence is a product of chances; you can't expect the > unexpected to happen too often in a row. Of course. I specified it to highlight two things: - You can't judge a simulation using just one or few real matches as benchmark. - If a simulation has Chelsea at 70 points and Chelsea really has 69 points you can't just say that the simulation is wrong. (this was functional to my idea to deduce teams' strength from the real table, idea that thank to you now I know is wrong) > - the simulation has the advantage that it can be repeated and its > statistics checked out. If and when you have time I just have to develop the habit to start a new simulation before I go to sleep, and the morning after I'll find the final table (indeed I guess that it doesn't take more than a couple of hours to complete a season, probably less). Write down the number of points for each team, and it's done. (tl;dr: it takes no time) > please make several > runs How much is "several"? > and check the following things: > 1. How similar to each other the simulated tables are at the end of > the season. This will tell us how consistently the FM engine applies the > parameters of the players and teams. If the tables turn out to be more > or less similar (with each team moving by no more than, let's say, 2 to > 3 positions around the table), we can conclude that it is a consistent > engine. What's the metric? Variance of number of points (or: of position) for each team? > 3. And of course we can see how it is influenced by randomness, by > looking at the outcomes of one match played several times. This is > allowed to have a bigger variance, but still within limits. One wouldn't > expect ManU to beat Portsmouth less than 8 times out of 10. Hmm... If it's just *one* match each season (i.e. if every morning I'll have to check just ManU vs. Portsmouth), it's OK. Otherwise it's not practical, because there isn't (AFAIK) a way to export results from FM to a spreadsheet. > - what you say below, is backwards. Thanks for the explanation! > We can, however, > calculate the statistics of the simulation, and then see if the real > results fall within the predicted confidence intervals. And that will > tell us how true to life the simulation really is - if the predicted > points per game for ManU is x points, and the actual one is y, we can > check to see within which confidence interval of the simulation that > value falls, and deduce the quality of the simulation. I.e. after several runs of the simulation we can say that we know each team' strength (as measured by points per match) according to FM (parameter), and we check how many standard errors separate the real EPL table (sample) from it. I.e. we ask: how likely is that the real table is generated by FM? Am I right? Sorry for questionable lexicon: all my statistical knowledge comes from my current reading of Statistics for Dummies*... and I'm understanding just half of what I am reading! * Really! I can't keep annoying Daniele for every little doubt, after all. -- Cheers milivella
From: milivella on 26 Mar 2010 03:52
milivella: > 2nd update: October 16: Fulham is still 18th. 3rd update: November 16: Fulham is 15th. Data-filled screenshots: Played matches: http://i.imgur.com/dj1ai.jpg Competitions: - Premier League: http://i.imgur.com/5qfzS.jpg - Europa League: http://i.imgur.com/tDHBS.jpg - [out of League Cup] Players' stats: - Goalkeepers: http://i.imgur.com/XGGz2.jpg - Defenders: http://i.imgur.com/9J7br.jpg - Midfielders: http://i.imgur.com/f5UZY.jpg [this URL is an evident homage to the Mothers of Invention] - Attackers: http://i.imgur.com/ZeHDb.jpg -- Cheers milivella |