nuplan-devkit icon indicating copy to clipboard operation
nuplan-devkit copied to clipboard

Inconsistent scenario types returned

Open muety opened this issue 10 months ago • 1 comments

I might have fundamentally misunderstood the concept of "scenario types", but the following behavior seems odd to me.

# filter by 'starting_unprotected_cross_turn'
f1 = ScenarioFilter(
    scenario_types=['starting_unprotected_cross_turn'],
    scenario_tokens=None,
    log_names=None,
    map_names=None,
    num_scenarios_per_type=None,
    limit_total_scenarios=None,
    timestamp_threshold_s=None,
    ego_displacement_minimum_m=None,
    expand_scenarios=False,
    remove_invalid_goals=False,
    shuffle=False,
)

scenarios = builder.get_scenarios(scenario_filter=f1, worker=SingleMachineParallelExecutor())

s = [s for s in scenarios if s.token == '454B09876AC457D2'.lower()][0]
print(s.token)  # -> 454b09876ac457d2
print(s.scenario_type)  # -> starting_unprotected_cross_turn

However:

# filter by token
f1 = ScenarioFilter(
    scenario_types=None,
    scenario_tokens=[t.lower() for t in ['454B09876AC457D2']],
    log_names=None,
    map_names=None,
    num_scenarios_per_type=None,
    limit_total_scenarios=None,
    timestamp_threshold_s=None,
    ego_displacement_minimum_m=None,
    expand_scenarios=False,
    remove_invalid_goals=False,
    shuffle=False,
)

scenarios = builder.get_scenarios(scenario_filter=f1, worker=SingleMachineParallelExecutor())

s = [s for s in scenarios if s.token == '454B09876AC457D2'.lower()][0]
print(s.token)  # -> 454b09876ac457d2
print(s.scenario_type)  # -> traversing_traffic_light_intersection

Why does the same scenario (given its token) have a different type depending on how I define the filters?!

muety avatar Feb 18 '25 10:02 muety

Debugging this led me to the following line where one of multiple tags is arbitrarily selected:

https://github.com/motional/nuplan-devkit/blob/d60b4cd2071de9bb041509c43f5226dd22f248c0/nuplan/database/nuplan_db/nuplan_scenario_queries.py#L888

If a type filter is applied, the returned set of types will only return a single entry in the first place, which is why we end up with two different results:

https://github.com/motional/nuplan-devkit/blob/d60b4cd2071de9bb041509c43f5226dd22f248c0/nuplan/database/nuplan_db/nuplan_scenario_queries.py#L812-L818

This is super confusing in two ways: first, why is the scenario_type property of NuplanScenario a single string in the first place, instead of a list, if a scenario can have multiple types? And, second, the devkit should at least set that type consistently.

muety avatar Feb 18 '25 10:02 muety