Please support us by disabling your adblock on our domain!

How random are the tests results?



  • First of all, thank you very much for setting up this page and testing the tactics. It's really useful for kinda lazy players like me and also interesting to see what does the ME favour (wingbacks are awesome, but strikerless seems to be unintuitive or broken).

    I've been wondering how random are the results though. Your methodology with setting up the league looks great and I see that the results should be consistent as they rely on testing across a whole season, but still I'm wondering 🙂

    Have you by chance run the same tactic for a few runs, to see how different are the results in each run?


  • Admin

    @gtd84 We did everything you could possibly do today (FM Editor, In-game editor, FMRTE, FM Fitness Coach, etc.) to make the tests results as accurate as possible. And each tactics gets tested twice (once with a sub-top team, once with an underdog). Anyhow, if you tested each tactic twice there would probably be a little difference that is just because of the randomness in Football Manager and we can't really fix that.

    Check this out: @MrLangvatn said in Give all 5 star tactics another round of test:

    @pwolf From my experience though, the difference in points is usually like under 10 points in the re-tests we have done before. And, even if we made 100x re-tests, the nr 1 tactic would still not be "the best" for all teams. It would just be the best in our test.. In personal savegames, people have different players, and they will fit different tactics. I would thus not distinguish in quality between any of the 5-star tactics, as there really is no way to know which is "best" in all leagues for all teams. That said, it is fairly safe to presume that a 5-star tactic in our test will outperform the 4-star tactics for most teams. But, many of the 4,5 star tactics will fit many teams/leagues better than the 5-star ones and will thus outperform them in the right circumstances.

    So, it wouldn't be fair to any of the tacticians to "crown" a winner....



  • Thanks for the info. Good to know that I should expect 10 point difference but @pwolf also made some good points. I was looking at the underdog/subtop results as if some tactics were more demanding and some easier for weaker teams, which is probably wrong 🙂

    I wonder if it's happening because of general FM randomness or there are even more variables that Fitness Coach and Test Leagues can't cover, like some tactics being more vulnerable to weather or strict referees. I'd bet it is because of randomness, so for me it would be interesting to see a run of 5 tests with the same team and tactics, just to see how it plays out. Of course testing new tactics is more interesting and important, and I'm very happy you're doing that!


  • Admin

    @gtd84 I agree with you that there is a factor in our test that is some randomness but as I said you made sure that it was as low as it could possibly be. If we could talk to Miles and make @Giggiux become a developer of the game then I'm sure he could create the perfect tactic testing system 😃 For right now, our tests are the most accurate you can find 😉


  • Admin

    @Fra I don't think they'll like to have "the perfect testing system" 😛


  • Admin

    @Giggiux said in How random are the tests results?:

    @Fra I don't think they'll like to have "the perfect testing system" 😛

    True 😃



  • I think something is not right in the latest testing. Look at this 4231 my favorite tactic in 17.2* vs 17.3*

    17.2* only 6 points difference in a repeated test
    http://forum.mrltactics.com/topic/219
    http://forum.mrltactics.com/topic/315

    17.3* has 39 points difference!
    http://forum.mrltactics.com/topic/674/iwb-4231
    http://forum.mrltactics.com/topic/672/iwb-4231


  • Admin

    @pwolf 39 points seems abit high yeah. It COULD happen though, if everything we can't control went against the test team. It's kind of like when you hit 15 red in a row in roulette. It does happen, but very rarely. Anyways, at this time, there is nothing more we can do to further reduce randomness without extensively widening our test to even more matches (which is something we might do for FM18).

    But, to be sure everything is right in the test, maybe @Giggiux can double check those saves and see if they really are the same tactic, and if everything looks good in both of them (maybe some bug with fm fitness coach happened or something).


  • Admin

    Ehm, I deleted the 17.2 saves, but I can check if the tactic and the OIs are the same and if Fitness Coach worked properly 🙂 in the new save



  • I'm not suggesting widening the test, just running 1 average tactics 5 times, to see how big is the random factor 🙂



  • @MrLangvatn I agree that could happen but in the latest test the repeated test results for few tactics all have way more points difference than in both 17.1* and 17.2* test so I think it's worth to investigate. One more round of retest for all 5 star tactics can further confirm.


 

You may also be interested in:

Please consider donating us if you like our service. The donations will be invested into the site to make it better.

Looks like your connection to MrLTactics.com was lost, please wait while we try to reconnect.