Bugs in multiagent model

[boqiang-li]

Bug Description

I think in the multiagent model, the function policy_mapping_fn may have some bugs or the tutorial 14_multiagent might have something wrong. I build the non-shared policy multiagent model. I have two kind of model:

1. For all of the traffic light in the network. And using shared model for all of the traffic light.
2. For all RL vehicle in the network, and using shared model for all of the RL vehicle.

And my code is as this:

from flow.networks import TrafficLightGridNetwork

...

vehicles.add(
    veh_id="rl",
    acceleration_controller=(RLController, {}),
    routing_controller=(GridRouter, {}),
    num_vehicles=num_vehicles)
  
inflow.add(
    veh_type="rl",
    edge=edge,
    vehs_per_hour=RL_INFLOW_RATE,
    departLane="free",
    departSpeed=V_ENTER,
    name="rlveh")
def tl_policy():
    """Generate a policy in RLlib."""
    config_tl={'gamma': 0.997}
    return PPOTFPolicy, obs_space_tl, act_space_tl, config_tl
    
def veh_policy():
    """Generate a policy in RLlib."""
    config_veh={'gamma': 0.998}
    return PPOTFPolicy, obs_space_veh, act_space_veh, config_veh
  
# Setup PG with a single policy graph for all agents
POLICY_GRAPHS = {'trafficlight': tl_policy(),'veh': veh_policy()}

def policy_mapping_fn(agent_id):
    print("------agent_id------")
    print(agent_id)
    if agent_id.startswith("rlveh"):
        return "veh"
    if agent_id.startswith("center"):
        return "trafficlight"
def get_state(self):
    ....
    return {'trafficlight': obs_tl, 'veh': obs_veh}
def compute_reward(self, rl_actions, **kwargs):
    ...
    return {'trafficlight': rews_tl, 'veh': rews_veh}

Bug Reproduce

As you can see, I use the TrafficLightGridNetwork, so the name of the center with traffic light should be 'centerX', like 'center0', 'center1'.

And the name of RL vehicles should be 'rlvehX'.

In the input of the function policy_mapping_fn, I think the 'agent_id' should be the name of the agent, like 'centerX' and 'rlvehX'.

BUT! When I define the return of the function get_state, there are somethings strange, when I run my code, the input of the policy_mapping_fn, all of the 'agent_id' is 'trafficlight'!

What's more, when I change the return of the get_state, such as

def get_state(self):
    ....
    return {'tl': obs_tl, 'veh': obs_veh}

all of the 'agent_id' is changed to 'tl'!

Is there something wrong with my codes?

The exact command that you did run that caused the bug: command

Here are my source code:

maincode.py

environment.py

Please rename the environment file and put it into the Env list. Then you can run it by

python maincode.py