Okay, I simplified it to just:
'meta_query' => array( 'relation' => 'OR',
array( 'key' => 'game_home_team', 'value' => $team_id ),
array( 'key' => 'game_away_team', 'value' => $team_id )
)
Ignoring the sport and season. And I am still only getting home or away games depending on the order of the keys.