Weird behavior of ScenarioFilter

[mh0797]

Hi, I noticed some counter-intuitive behavior of the ScenarioFilter.

I used the following minimal example to reproduce the issue:

import os
import hydra
from hydra.utils import instantiate
from nuplan.planning.scenario_builder.nuplan_db.nuplan_scenario_builder import NuPlanScenarioBuilder
from nuplan.planning.scenario_builder.scenario_filter import ScenarioFilter
from nuplan.planning.utils.multithreading.worker_pool import WorkerPool


path_to_devkit = os.path.relpath('/path/to/nuplan/devkit', os.getcwd())

hydra.core.global_hydra.GlobalHydra.instance().clear()
hydra.initialize(config_path=os.path.join(path_to_devkit, "nuplan/planning/script/config/common/scenario_builder"))
cfg = hydra.compose(config_name='nuplan_mini')
scenario_builder:NuPlanScenarioBuilder = instantiate(cfg)

hydra.core.global_hydra.GlobalHydra.instance().clear()
hydra.initialize(config_path=os.path.join(path_to_devkit, "nuplan/planning/script/config/common/worker"))
cfg = hydra.compose(config_name='ray_distributed')
worker:WorkerPool = instantiate(cfg)

hydra.core.global_hydra.GlobalHydra.instance().clear()
hydra.initialize(config_path=os.path.join(path_to_devkit, "nuplan/planning/script/config/common/scenario_filter"))
cfg = hydra.compose(config_name='training_scenarios')
scenario_filter:ScenarioFilter = instantiate(cfg)
scenarios = scenario_builder.get_scenarios(scenario_filter, worker)
print('NO FILTER: found', len(scenarios), 'scenarios in total')
scenario_types = set([scenario.scenario_type for scenario in scenarios])
print('NO FILTER: scenario_types', scenario_types)

available_types = scenario_builder._db.get_all_scenario_types()
hydra.core.global_hydra.GlobalHydra.instance().clear()
hydra.initialize(config_path=os.path.join(path_to_devkit, "nuplan/planning/script/config/common/scenario_filter"))
cfg = hydra.compose(config_name='training_scenarios', overrides=[
    f"scenario_types={available_types}",
])
scenario_filter:ScenarioFilter = instantiate(cfg)
scenarios = scenario_builder.get_scenarios(scenario_filter, worker)
print('FILTER AVAILABLE TYPES: found', len(scenarios), 'scenarios in total')
scenario_types = set([scenario.scenario_type for scenario in scenarios])
print('FILTER AVAILABLE TYPES: scenario_types', scenario_types)

I get the following output

NO FILTER: found 511215 scenarios in total
NO FILTER: scenario_types {'unknown'}`

FILTER AVAILABLE TYPES: found 17100 scenarios in total
FILTER AVAILABLE TYPES: scenario_types {'changing_lane', 'on_stopline_crosswalk', 'near_long_vehicle', 'stationary_at_crosswalk', 'starting_high_speed_turn', 'accelerating_at_traffic_light', 'stationary', 'on_stopline_traffic_light', 'near_pedestrian_on_crosswalk_with_ego', 'near_barrier_on_driveable', 'stationary_at_traffic_light_with_lead', 'following_lane_without_lead', 'near_pedestrian_on_crosswalk', 'on_pickup_dropoff', 'stopping_at_stop_sign_with_lead', 'stopping_at_traffic_light_with_lead', 'starting_straight_stop_sign_intersection_traversal', 'starting_low_speed_turn', 'accelerating_at_crosswalk', 'high_magnitude_speed', 'starting_left_turn', 'near_multiple_pedestrians', 'traversing_intersection', 'traversing_pickup_dropoff', 'stopping_with_lead', 'high_lateral_acceleration', 'on_traffic_light_intersection', 'near_pedestrian_at_pickup_dropoff', 'starting_protected_noncross_turn', 'behind_bike', 'near_high_speed_vehicle', 'stationary_at_traffic_light_without_lead', 'on_all_way_stop_intersection', 'high_magnitude_jerk', 'near_multiple_vehicles', 'changing_lane_to_left', 'accelerating_at_stop_sign_no_crosswalk', 'starting_straight_traffic_light_intersection_traversal', 'near_trafficcone_on_driveable', 'stationary_in_traffic', 'accelerating_at_traffic_light_without_lead', 'behind_pedestrian_on_pickup_dropoff', 'waiting_for_pedestrian_to_cross', 'accelerating_at_stop_sign', 'starting_unprotected_noncross_turn', 'behind_long_vehicle', 'near_construction_zone_sign', 'on_stopline_stop_sign', 'following_lane_with_lead', 'low_magnitude_speed', 'following_lane_with_slow_lead', 'starting_protected_cross_turn', 'stopping_at_stop_sign_no_crosswalk', 'accelerating_at_traffic_light_with_lead', 'starting_right_turn', 'traversing_crosswalk', 'stopping_at_traffic_light_without_lead', 'starting_unprotected_cross_turn', 'traversing_traffic_light_intersection', 'stopping_at_crosswalk', 'on_carpark', 'traversing_narrow_lane', 'stopping_at_stop_sign_without_lead', 'changing_lane_to_right', 'medium_magnitude_speed', 'on_intersection'}

For the first case (no filter) the scenario filter works as expected yielding all 511.215 scenarios in the nuplan_mini dataset. Limiting the total number of scenarios works as expected as well (not in example). However, when filtering scenarios by types, the number of scenarios is only 17.100 even though all available types are selected. I am not sure if this is a bug. Can you clarify why this happens?

[patk-motional]

Hi @mh0797,

Sorry for the delay in replying! We've just updated the devkit again to v0.4. Can you re-try with the newest master?

[mh0797]

I did the same check again (slightly different code since scenario_builder._db.get_all_scenario_types() does not exist anymore). I found out that only ~10k samples are tagged with a scenario_type while all the others are of type unknown. Is this a bug or are most scenarios just unlabeled?

[dimitris-motional]

Hi @mh0797,

This is expected, a lot of scenarios are not tagged as they don't belong to any of the available types. In addition, many scenarios are sparsely tagged to reduce tag overlap - we'll be releasing more dense tags in the next dataset version very soon.

[gianmarco-motional]

I'll close the issue for inactivity. If you still are facing the problem with the current devkit version, feel free to reopen it!