Away Agent seems win more than Home Agent

[Drefsante]

Tested with 'scripted' bot (but also another bot) and difference is really important for high number of games. Seems not related to competition_mode, pathfinding or debug_mode. For example from my tests (each line correspond to a run of 500 games):

Home Win	Away Win
102	236
102	239
115	225
116	219
122	226
125	216

Here is the script I have used:

config = botbowl.load_config("bot-bowl")
config.competition_mode = False
config.pathfinding_enabled = False
config.debug_mode = False

ruleset = botbowl.load_rule_set(config.ruleset, all_rules=True)
arena = botbowl.load_arena(config.arena)

home = botbowl.load_team_by_filename("human", ruleset)
away = botbowl.load_team_by_filename("human", ruleset)

results = []
win_home = 0
loss_home = 0

home_agent = botbowl.make_bot('scripted')
home_agent.name = 'scripted_Home'

away_agent = botbowl.make_bot('scripted')
away_agent.name = 'scripted_Away';

nGames = 500
for i in range(nGames):
    game = botbowl.Game(i, home, away, home_agent, away_agent, config, arena=arena, ruleset=ruleset, record=False)   
    game.init()

    results.append(str(game.state.home_team.state.score) + "-" + str(game.state.away_team.state.score))
    
    if game.state.home_team.state.score > game.state.away_team.state.score:
        win_home = win_home + 1
    if game.state.home_team.state.score 

[njustesen]

This is interesting!

This suggests that we have a bug in the scripted bot, in the pathfinding module, or in the game implementation. We could run similar tests with the random bot on smaller boards to see if observe the same skewed results.

[Drefsante]

Yes I tested yesterday on smaller boards and it was ok. So it's only with the classical board. And I have same behavior with my bot (but I did less tests because it's slower and difference was less obvious), so I think problem is not in scripted bot. And problem is there even without pathfinding activated. So it looks it's in game implementation but no idea where...

[njustesen]

So it's only with the classical board.

The random bots can't win on the classical board so I guess we don't know for sure?

And problem is there even without pathfinding activated.

Cool. Then it's probably not the pathfinding.

So it looks it's in game implementation but no idea where...

If you look at the MCTS tutorial, home wins more that away because when in doubt of which action to take, it will default to the first action in the list which will move it towards top-left, i.e. the away team's endzone, resulting in occasional touchdowns. It's thus interesting that you see the opposite pattern here where away is winning.

[mrbermell]

Must be a very subtle thing since both sides win the game. Here are some thoughts:

• Shouldn't that script always generate the same result? Since the seed is specified (Game(i, ....)) and the scripted bot doesn't have any stochastic elements. Hmm...
• As njustesen mentioned, here and there are small artifacts that could sometimes benefit one side over the other. An example is how the scripted bot chooses the push square, it always picks the first position. Could this benefit one side over the other?

def push(self, game):
    for position in game.state.available_actions[0].positions:
        return Action(ActionType.PUSH, position=position)

• Don't be too quick to rule out the pathfinding module, the scripted bot is still using it. Would be interesting to see if we can replicate the results with a decently trained RL bot that doesn't use pathfinding at all.

[Drefsante]

I have done a 500 runs with my own scripted bot (which is not using pathfinding), and I get: Home: 159 wins Away: 200 wins

Another 500 runs: Home: 156 wins Away: 203 wins

Difference is less obvious than 'scripted' bot but still there. (notice I also tried to set a random seed for each game but it has no impact)

[mrbermell]

That should then rule out the pathfinding and the scripted bot. One way to narrow the search is to do a frequency analysis of the outcomes.

[Niaamo8]

I think I may have found a little related bug in the scripted bot.

In the part "9. Move towards the ball" there's two occurrences of: if shortest_distance is None or (p.prob == 1 and distance < shortest_distance):

Shouldn't these rather be: if p.prob == 1 and (shortest_distance is None or distance < shortest_distance):

Path finding returns the paths ordered so that paths ending most to the left are first. In the current implementation if the first path (that takes the player left) ends up at least as close to the ball than any non-risky path in the list, the first path ends up chosen, however risky it is. This will hurt the home team more, as the home team attacks from right to left, so it's more likely for it to end up making very risky dodges through the lines towards the left.

Testing this fix only resulted in a minor improvement in the balance though, so there must be something further that favors the away side.

I ran two tests of 500:

Home team Wins: 127 Home team Losses: 195

Home team Wins: 141 Home team Losses: 206

[Drefsante]

Ok so I'm not the only one to reproduce the issue, that's interesting :-) Notice that if this bug is not resolved for Bot Bowl tournament, it should be needed to do same number of games at home and away. So each bot should face each other 12 games (not 10)... (because I think bug is not - only - in scripted bot but could be in core)

[njustesen]

Notice that if this bug is not resolved for Bot Bowl tournament, it should be needed to do same number of games at home and away. So each bot should face each other 12 games (not 10)...

Why 12 instead of 10?

[Drefsante]

Why 12 instead of 10?

Because I'm stupid :-) 10 is fine!