nuplan-devkit icon indicating copy to clipboard operation
nuplan-devkit copied to clipboard
verified publisher icon motional

Nuboard not displaying any information

Open kensukenk opened this issue 3 years ago • 23 comments

Hi! When I run the nuplan_framework.ipynb notebook, I can get all the models to train but not shows up when I launch Nuboard. I get some messages when I train and simulate that maybe suggest the error?

"The agent on node Blade-15 failed to be restarted 5 times. There are 3 possible problems if you see this error.

  1. The dashboard might not display correct information on this node.
  2. Metrics on this node won't be reported.
  3. runtime_env APIs won't work. Check out the dashboard_agent.log to see the detailed failure messages." dashboard_agent.log

This is what appears when I try to launch Nuboard. It seems that there are two experiments, but no data shows up at all. Screenshot from 2022-06-27 11-37-48

I know that Nuboard is under development, but I was wondering if I was supposed to be able to see anything at all? If not, where would I be able to find the raw data from simulation and training?

kensukenk avatar Jun 27 '22 18:06 kensukenk

Hi @kensukenk,

Yes, you should be able to see something. I will ask my colleague @kokseang-motional to take a look. In the meantime, can you share the command you used to

  1. Run the simulation
  2. Run nuBoard

patk-motional avatar Jun 28 '22 00:06 patk-motional

Hey @kensukenk Would u mind sharing the commands u run simulation and nuboard? Also, did u see any error messages in cli? Did overview and scenario tabs output sth? Thanks for reporting this

kokseang-motional avatar Jun 28 '22 01:06 kokseang-motional

I just ran jupyter notebook ~/nuplan-devkit/tutorials/nuplan_framework.ipynb in my terminal and then ran every cell down the notebook. Nothing outputs in any tab in nuboard. This is my version of the notebook in case I accidentally changed anything but I don't believe I did: nuplan_framework.zip

I just installed nuplan yesterday, so to be clear, in my home directory I have both ~/nuplan-devkit and ~/nuplan? Screenshot from 2022-06-27 21-21-25 Screenshot from 2022-06-27 21-21-14

kensukenk avatar Jun 28 '22 04:06 kensukenk

Hi @kensukenk Can u try changing this line f'simulation_path={[nuboard_file_1, nuboard_file_2]}' to f'simulation_path={[nuboard_file_1]}' only?

kokseang-motional avatar Jul 01 '22 06:07 kokseang-motional

I still have the same issue only using nuboard_file_1. In case this is useful, here are the aggregated metrics for the simulation simple experiment. metrics.csv

kensukenk avatar Jul 01 '22 18:07 kensukenk

Hi @kensukenk What happened if u run simulation in cli and then nuboard? Does it show results correctly?

kokseang-motional avatar Jul 04 '22 03:07 kokseang-motional

It still does not work. Does it matter that I get certain runtime warnings such as "RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887202) diff_b_a = subtract(b, a)" or "(raylet) E0704 12:01:28.406616966 887721 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers"?

kensukenk avatar Jul 04 '22 19:07 kensukenk

@kensukenk can you post your full stack trace?

patk-motional avatar Jul 05 '22 02:07 patk-motional

This is the stack trace for simulation:

python nuplan/planning/script/run_simulation.py \

+simulation=open_loop_boxes \
planner=simple_planner \
scenario_builder=nuplan_mini \
scenario_filter=all_scenarios \
scenario_filter.scenario_types="[near_multiple_vehicles, on_pickup_dropoff, starting_unprotected_cross_turn, high_magnitude_jerk]" \
scenario_filter.num_scenarios_per_type=10 \

/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. warnings.warn( Global seed set to 0 INFO:nuplan.planning.script.builders.main_callback_builder:Building MultiMainCallback... INFO:nuplan.planning.script.builders.main_callback_builder:Building MultiMainCallback: 3...DONE! 2022-07-04 12:01:04,158 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:19} Building WorkerPool... 2022-07-04 12:01:05,487 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_ray.py:72} Starting ray local! 2022-07-04 12:01:07,888 INFO services.py:1374 -- View the Ray dashboard at http://10.23.245.54:8265 2022-07-04 12:01:10,061 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:100} Worker: RayDistributed 2022-07-04 12:01:10,061 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:101} Number of nodes: 1 Number of CPUs per node: 16 Number of GPUs per node: 1 Number of threads across all nodes: 16 2022-07-04 12:01:10,062 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:27} Building WorkerPool...DONE! 2022-07-04 12:01:10,062 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:29} Building experiment folders... 2022-07-04 12:01:10,062 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:32}

Folder where all results are stored: /home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03

2022-07-04 12:01:10,063 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:58} Building experiment folders...DONE! 2022-07-04 12:01:10,063 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:18} Building AbstractScenarioBuilder... (raylet) E0704 12:01:12.877570964 887631 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers 2022-07-04 12:01:12,998 INFO {/home/kensuke/nuplan-devkit/nuplan/database/nuplan_db/nuplandb_wrapper.py:166} Loaded maps DB 2022-07-04 12:01:15,751 INFO {/home/kensuke/nuplan-devkit/nuplan/database/nuplan_db/nuplandb_wrapper.py:176} Loaded 72 log DBs 2022-07-04 12:01:15,765 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:21} Building AbstractScenarioBuilder...DONE! 2022-07-04 12:01:15,766 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_callback_builder.py:24} Building AbstractCallback... 2022-07-04 12:01:15,773 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_callback_builder.py:36} Building AbstractCallback: 2...DONE! 2022-07-04 12:01:15,773 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:45} Building simulations... 2022-07-04 12:01:15,773 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:19} Building ScenarioFilter... 2022-07-04 12:01:15,775 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:22} Building ScenarioFilter...DONE! 2022-07-04 12:01:15,775 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:52} Extracting scenarios... (raylet) E0704 12:01:18.203487316 887706 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers 2022-07-04 12:01:18,310 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:54} Extracting scenarios 34...DONE! 2022-07-04 12:01:18,319 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:58} Building metric engines... 2022-07-04 12:01:18,800 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:60} Building metric engines...DONE 2022-07-04 12:01:18,807 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:64} Building simulations from 34 scenarios... 2022-07-04 12:01:20,083 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/simulation_builder.py:112} Building simulations...DONE! 2022-07-04 12:01:20,083 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/run_simulation.py:75} Running simulation... 2022-07-04 12:01:20,083 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/utils.py:128} Executing runners... 2022-07-04 12:01:20,083 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/runner/executor.py:83} Starting 34 simulations using RayDistributed! Ray objects: 0%| | 0/34 00:00<?, ?it/s /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887197) warnings.warn( (pid=887207) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887207) warnings.warn( (pid=887195) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887195) warnings.warn( (pid=887206) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887206) warnings.warn( (pid=887200) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887200) warnings.warn( (pid=887196) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887196) warnings.warn( (pid=887201) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887201) warnings.warn( (pid=887190) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887190) warnings.warn( (pid=887198) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887198) warnings.warn( (pid=887199) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887199) warnings.warn( (pid=887204) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887204) warnings.warn( (pid=887202) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887202) warnings.warn( (pid=887193) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887193) warnings.warn( (pid=887191) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887191) warnings.warn( (pid=887192) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887192) warnings.warn( (pid=887194) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=887194) warnings.warn( (raylet) E0704 12:01:28.406616966 887721 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers (raylet) E0704 12:01:47.294714735 887731 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887201) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887201) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887201) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887201) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887201) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887201) diff_b_a = subtract(b, a) (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 3%|▉ | 1/34 [00:57<31:48, 57.84s/it](wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887202) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887202) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887202) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887202) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887193) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887193) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887193) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887193) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (raylet) E0704 12:02:22.522624360 887781 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers 2022-07-04 12:02:23,239 WARNING worker.py:1257 -- (ip=10.23.245.54) The agent on node Blade-15 failed to be restarted 5 times. There are 3 possible problems if you see this error.

  1. The dashboard might not display correct information on this node.
  2. Metrics on this node won't be reported.
  3. runtime_env APIs won't work. Check out the dashboard_agent.log to see the detailed failure messages. (wrapped_fn pid=887204) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887204) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887204) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887204) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887202) WARNING:root:Goal is not specified, skipping ego_relative_progress_to_goal metric (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887207) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887207) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887207) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887207) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887190) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887190) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887190) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887190) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887194) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887194) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887194) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887194) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887199) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887199) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887199) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887199) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887206) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887206) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887206) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887206) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887198) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887198) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887198) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887198) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887191) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887191) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887191) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887191) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887197) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887197) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887197) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887197) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887200) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887200) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887200) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887200) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887202) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887202) diff_b_a = subtract(b, a) (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 6%|█▊ | 2/34 [01:13<17:43, 33.23s/it](wrapped_fn pid=887193) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887193) diff_b_a = subtract(b, a) (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 9%|██▋ | 3/34 [01:15<09:36, 18.59s/it](wrapped_fn pid=887204) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887204) diff_b_a = subtract(b, a) (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 12%|███▌ | 4/34 [01:22<07:01, 14.03s/it](wrapped_fn pid=887199) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887199) diff_b_a = subtract(b, a) (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 15%|████▍ | 5/34 [01:23<04:32, 9.41s/it](wrapped_fn pid=887198) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887198) diff_b_a = subtract(b, a) (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 18%|█████▎ | 6/34 [01:23<02:56, 6.31s/it](wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887196) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887196) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887196) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887196) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887206) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887206) diff_b_a = subtract(b, a) (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 21%|██████▏ | 7/34 [01:26<02:18, 5.13s/it](wrapped_fn pid=887190) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887190) diff_b_a = subtract(b, a) (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 24%|███████ | 8/34 [01:27<01:41, 3.91s/it](wrapped_fn pid=887200) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887200) diff_b_a = subtract(b, a) (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! (wrapped_fn pid=887197) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887197) diff_b_a = subtract(b, a) (wrapped_fn pid=887194) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887194) diff_b_a = subtract(b, a) Ray objects: 26%|███████▉ | 9/34 [01:28<01:16, 3.07s/it](wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 29%|████████▌ | 10/34 [01:28<00:51, 2.16s/it](wrapped_fn pid=887191) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887191) diff_b_a = subtract(b, a) (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 35%|██████████▏ | 12/34 [01:29<00:30, 1.40s/it](wrapped_fn pid=887207) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887207) diff_b_a = subtract(b, a) (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 38%|███████████ | 13/34 [01:30<00:25, 1.22s/it](wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887192) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887192) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887192) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887192) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887196) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887196) diff_b_a = subtract(b, a) (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 41%|███████████▉ | 14/34 [01:47<01:46, 5.34s/it](wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887195) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887195) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887195) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887195) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887192) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887192) diff_b_a = subtract(b, a) (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 44%|████████████▊ | 15/34 [02:19<03:58, 12.57s/it](wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887199) WARNING:root:Goal is not specified, skipping ego_relative_progress_to_goal metric (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887201) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887201) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887201) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887201) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887195) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887195) diff_b_a = subtract(b, a) (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 47%|█████████████▋ | 16/34 [02:43<04:45, 15.85s/it](wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887199) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887199) diff_b_a = subtract(b, a) (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 50%|██████████████▌ | 17/34 [02:46<03:24, 12.03s/it](wrapped_fn pid=887194) WARNING:root:Goal is not specified, skipping ego_relative_progress_to_goal metric (wrapped_fn pid=887193) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887193) diff_b_a = subtract(b, a) (wrapped_fn pid=887204) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887204) diff_b_a = subtract(b, a) (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 53%|███████████████▎ | 18/34 [02:48<02:26, 9.17s/it](wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887204) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 56%|████████████████▏ | 19/34 [02:48<01:37, 6.53s/it](wrapped_fn pid=887194) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887194) diff_b_a = subtract(b, a) (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887194) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 59%|█████████████████ | 20/34 [02:51<01:18, 5.62s/it](wrapped_fn pid=887200) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887200) diff_b_a = subtract(b, a) (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887200) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 62%|█████████████████▉ | 21/34 [02:52<00:52, 4.03s/it](wrapped_fn pid=887202) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887202) diff_b_a = subtract(b, a) (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887202) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 65%|██████████████████▊ | 22/34 [02:52<00:36, 3.07s/it](wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887207) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887207) diff_b_a = subtract(b, a) (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887207) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 68%|███████████████████▌ | 23/34 [02:56<00:36, 3.36s/it](wrapped_fn pid=887206) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887206) diff_b_a = subtract(b, a) (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887206) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 71%|████████████████████▍ | 24/34 [03:01<00:36, 3.66s/it](wrapped_fn pid=887190) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887190) diff_b_a = subtract(b, a) (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887190) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 74%|█████████████████████▎ | 25/34 [03:02<00:25, 2.88s/it](wrapped_fn pid=887198) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887198) diff_b_a = subtract(b, a) (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887198) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 76%|██████████████████████▏ | 26/34 [03:02<00:16, 2.07s/it](wrapped_fn pid=887201) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887201) diff_b_a = subtract(b, a) (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887201) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 79%|███████████████████████ | 27/34 [03:03<00:11, 1.63s/it](wrapped_fn pid=887197) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887197) diff_b_a = subtract(b, a) (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887197) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 82%|███████████████████████▉ | 28/34 [03:05<00:11, 1.93s/it](wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887191) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887191) diff_b_a = subtract(b, a) (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887191) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 85%|████████████████████████▋ | 29/34 [03:09<00:11, 2.40s/it](wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887193) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887193) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887193) /home/kensuke/nuplan-devkit/nuplan/common/maps/nuplan_map/nuplan_map.py:403: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. (wrapped_fn pid=887193) self._vector_map['drivable_area'] = road_segments.append(intersections.append(generic_drivable_areas)).dropna( (wrapped_fn pid=887196) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887196) diff_b_a = subtract(b, a) (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887196) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 88%|█████████████████████████▌ | 30/34 [03:19<00:19, 4.84s/it](wrapped_fn pid=887192) WARNING:root:Goal is not specified, skipping ego_relative_progress_to_goal metric (wrapped_fn pid=887192) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887192) diff_b_a = subtract(b, a) (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887192) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 91%|██████████████████████████▍ | 31/34 [03:22<00:12, 4.08s/it](wrapped_fn pid=887193) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887193) diff_b_a = subtract(b, a) Ray objects: 94%|███████████████████████████▎ | 32/34 [03:25<00:07, 3.89s/it](wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887193) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887195) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887195) diff_b_a = subtract(b, a) (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887195) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 97%|████████████████████████████▏| 33/34 [03:30<00:04, 4.08s/it](wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Starting metrics computation... (wrapped_fn pid=887199) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/numpy/lib/function_base.py:4527: RuntimeWarning: invalid value encountered in subtract (wrapped_fn pid=887199) diff_b_a = subtract(b, a) (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Finished metrics computation! (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saving metric statistics! (wrapped_fn pid=887199) INFO:nuplan.planning.simulation.callback.metric_callback:Saved metrics! Ray objects: 100%|█████████████████████████████| 34/34 [03:38<00:00, 6.42s/it] 2022-07-04 12:04:58,396 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/runner/executor.py:100} Number of successful simulations: 34 2022-07-04 12:04:58,396 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/runner/executor.py:101} Number of failed simulations: 0 2022-07-04 12:04:58,396 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/utils.py:136} Finished executing runners! 2022-07-04 12:04:58,418 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/utils.py:64} Saved simulation reports to /home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03/simulations_report.parquet 2022-07-04 12:04:58,418 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/main_callback/time_callback.py:27} Simulation duration: 00:03:54 [HH:MM:SS] 2022-07-04 12:04:59,230 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/main_callback/metric_file_callback.py:55} Metric files integration: 00:00:00 [HH:MM:SS] 2022-07-04 12:04:59,355 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/main_callback/metric_aggregator_callback.py:45} Running metric aggregator: weighted_average_metric_aggregator 2022-07-04 12:04:59,416 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/simulation/main_callback/metric_aggregator_callback.py:53} Metric aggregator: 00:00:00 [HH:MM:SS] 2022-07-04 12:04:59,417 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/run_simulation.py:77} Finished running simulation!

kensukenk avatar Jul 05 '22 04:07 kensukenk

training:

python nuplan/planning/script/run_training.py experiment_name=raster_experiment py_func=train +training=training_raster_model scenario_builder=nuplan_mini scenario_filter.limit_total_scenarios=500 lightning.trainer.params.max_epochs=10 data_loader.params.batch_size=8 data_loader.params.num_workers=8 /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. warnings.warn( Global seed set to 0 2022-07-04 11:56:03,494 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:17} Building experiment folders... 2022-07-04 11:56:03,495 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/folder_builder.py:19} Experimental folder: /home/kensuke/nuplan/exp/exp/raster_experiment/2022.07.04.11.56.03 2022-07-04 11:56:03,495 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:19} Building WorkerPool... 2022-07-04 11:56:03,497 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_ray.py:72} Starting ray local! 2022-07-04 11:56:05,882 INFO services.py:1374 -- View the Ray dashboard at http://10.23.245.54:8265 2022-07-04 11:56:08,032 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:100} Worker: RayDistributed 2022-07-04 11:56:08,032 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/utils/multithreading/worker_pool.py:101} Number of nodes: 1 Number of CPUs per node: 16 Number of GPUs per node: 1 Number of threads across all nodes: 16 2022-07-04 11:56:08,032 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/worker_pool_builder.py:27} Building WorkerPool...DONE! 2022-07-04 11:56:08,033 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/training/experiments/training.py:42} Building training engine... 2022-07-04 11:56:08,033 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/model_builder.py:18} Building TorchModuleWrapper... 2022-07-04 11:56:08,609 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/timm/models/helpers.py:188} Loading pretrained weights from url (https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-rsb-weights/resnet50_a1_0-14fe96d1.pth) 2022-07-04 11:56:08,707 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/timm/models/helpers.py:208} Converted input conv conv1 pretrained weights from 3 to 4 channel(s) 2022-07-04 11:56:08,745 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/model_builder.py:21} Building TorchModuleWrapper...DONE! 2022-07-04 11:56:08,745 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/splitter_builder.py:18} Building Splitter... 2022-07-04 11:56:09,957 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/splitter_builder.py:21} Building Splitter...DONE! 2022-07-04 11:56:09,957 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:18} Building AbstractScenarioBuilder... (raylet) E0704 11:56:10.899484671 884432 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers 2022-07-04 11:56:12,854 INFO {/home/kensuke/nuplan-devkit/nuplan/database/nuplan_db/nuplandb_wrapper.py:166} Loaded maps DB 2022-07-04 11:56:15,850 INFO {/home/kensuke/nuplan-devkit/nuplan/database/nuplan_db/nuplandb_wrapper.py:176} Loaded 72 log DBs 2022-07-04 11:56:15,850 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_building_builder.py:21} Building AbstractScenarioBuilder...DONE! 2022-07-04 11:56:15,850 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:19} Building ScenarioFilter... 2022-07-04 11:56:15,852 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_filter_builder.py:22} Building ScenarioFilter...DONE! Ray objects: 0%| | 0/16 00:00<?, ?it/s E0704 11:56:16.699473880 884547 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers (pid=883990) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883990) warnings.warn( (pid=884003) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=884003) warnings.warn( (pid=884000) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=884000) warnings.warn( (pid=883999) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883999) warnings.warn( (pid=883993) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883993) warnings.warn( (pid=883989) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883989) warnings.warn( (pid=884005) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=884005) warnings.warn( (pid=883997) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883997) warnings.warn( (pid=883991) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883991) warnings.warn( (pid=884001) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=884001) warnings.warn( (pid=883995) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883995) warnings.warn( (pid=883996) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883996) warnings.warn( (pid=883992) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883992) warnings.warn( (pid=884004) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=884004) warnings.warn( (pid=883998) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883998) warnings.warn( (pid=883994) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. (pid=883994) warnings.warn( (raylet) E0704 11:56:27.614408888 884559 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers Ray objects: 50%|███████████████ | 8/16 00:29<00:10, 1.34s/it E0704 11:56:45.648005160 884581 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers Ray objects: 100%|█████████████████████████████| 16/16 [00:34<00:00, 2.17s/it] 2022-07-04 11:56:51,084 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/scenario_builder.py:151} Extracted 500 scenarios for training 2022-07-04 11:56:51,088 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/training_callback_builder.py:19} Building callbacks... 2022-07-04 11:56:51,106 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/builders/training_callback_builder.py:28} Building callbacks...DONE! GPU available: True, used: True TPU available: False, using: 0 TPU cores Using native 16bit precision. 2022-07-04 11:56:51,109 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/script/run_training.py:60} Starting training... Global seed set to 0 initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1 2022-07-04 11:56:51,114 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:194} Added key: store_based_barrier_key:1 to store for rank: 0 2022-07-04 11:56:51,115 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:224} Rank 0: Completed store-based barrier for 1 nodes.

distributed_backend=nccl All DDP processes registered. Starting ddp with 1 processes

2022-07-04 11:56:56,302 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/training/data_loader/datamodule.py:42} Number of samples in train set: 368 2022-07-04 11:56:56,314 INFO {/home/kensuke/nuplan-devkit/nuplan/planning/training/data_loader/datamodule.py:42} Number of samples in validation set: 57 LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

| Name | Type | Params

0 | model | RasterModel | 23.6 M

23.6 M Trainable params 0 Non-trainable params 23.6 M Total params 94.340 Total estimated model params size (MB) /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/pytorch_lightning/callbacks/lr_monitor.py:97: RuntimeWarning: You are using LearningRateMonitor callback with models that have no learning rate schedulers. Please see documentation for configure_optimizers method. rank_zero_warn( Epoch 0: 0%| | 0/53 [00:00<?, ?it/s]/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.) return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode) 2022-07-04 11:56:57,516 INFO {/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/torch/nn/parallel/distributed.py:788} Reducer buckets have been rebuilt in this iteration. Epoch 0: 87%|█████████████ | 46/53 [00:06<00:01, 6.75it/s, loss=231, v_num=]/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/pytorch_lightning/trainer/callback_hook.py:100: LightningDeprecationWarning: The signature of Callback.on_train_epoch_end has changed in v1.3. outputs parameter has been removed. Support for the old signature will be removed in v1.5 warning_cache.deprecation( Epoch 0: 89%|█████████████▎ | 47/53 [00:08<00:01, 5.37it/s, loss=231, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 1: 0%| | 0/53 [00:00<?, ?it/s, loss=231, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 1: 87%|█████████████ | 46/53 [00:08<00:01, 5.41it/s, loss=234, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) (raylet) E0704 11:57:19.226724981 885067 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) 2022-07-04 11:57:19,680 WARNING worker.py:1257 -- (ip=10.23.245.54) The agent on node Blade-15 failed to be restarted 5 times. There are 3 possible problems if you see this error.

  1. The dashboard might not display correct information on this node.
  2. Metrics on this node won't be reported.
  3. runtime_env APIs won't work. Check out the dashboard_agent.log to see the detailed failure messages. [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 2: 0%| | 0/53 [00:00<?, ?it/s, loss=234, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 2: 87%|█████████████ | 46/53 [00:07<00:01, 5.88it/s, loss=221, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 3: 0%| | 0/53 [00:00<?, ?it/s, loss=221, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 3: 87%|█████████████ | 46/53 [00:08<00:01, 5.61it/s, loss=199, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 4: 0%| | 0/53 [00:00<?, ?it/s, loss=199, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 4: 87%|█████████████ | 46/53 [00:08<00:01, 5.29it/s, loss=213, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 5: 0%| | 0/53 [00:00<?, ?it/s, loss=213, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 5: 87%|█████████████ | 46/53 [00:08<00:01, 5.21it/s, loss=233, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 6: 0%| | 0/53 [00:00<?, ?it/s, loss=233, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 6: 87%|█████████████ | 46/53 [00:08<00:01, 5.14it/s, loss=229, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 7: 0%| | 0/53 [00:00<?, ?it/s, loss=229, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 7: 87%|█████████████ | 46/53 [00:08<00:01, 5.33it/s, loss=241, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 8: 0%| | 0/53 [00:00<?, ?it/s, loss=241, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 8: 87%|█████████████ | 46/53 [00:08<00:01, 5.18it/s, loss=218, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 9: 0%| | 0/53 [00:00<?, ?it/s, loss=218, v_num=][W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 9: 87%|█████████████ | 46/53 [00:08<00:01, 5.15it/s, loss=209, v_num=[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) Epoch 9: 100%|███████████████| 53/53 [00:14<00:00, 3.68it/s, loss=209, v_num=]

kensukenk avatar Jul 05 '22 04:07 kensukenk

Hi @kensukenk,

To see the results of your simulation you need to launch nuBoard.

python nuplan/planning/script/run_nuboard.py

You will then be re-directed to your browser. From there you can upload the simulation file using the GUI. Your simulation should be here:

/home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03

patk-motional avatar Jul 05 '22 05:07 patk-motional

Even after uploading the simulation file no data shows up.

kensukenk avatar Jul 05 '22 06:07 kensukenk

Hey @kensukenk Would u mind zip and upload your simulation folder here? /home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03

I suspect there is no simulation data available in the folder

kokseang-motional avatar Jul 06 '22 01:07 kokseang-motional

The zipped file is too large to upload, so here is a google drive link with the zip file.

https://drive.google.com/file/d/1roBFx-T4cMZZqp4ucr7e8G5tEGy8Mg14/view?usp=sharing

kensukenk avatar Jul 06 '22 03:07 kensukenk

Hi @kensukenk Thanks for providing the zip file, it actually worked on my end Can u try the following command:

python nuplan/planning/script/run_nuboard.py simulation_path="[/home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03]"

kokseang-motional avatar Jul 08 '22 18:07 kokseang-motional

Hmmm, for some reason there is still no visualization loading on my end.

kensukenk avatar Jul 11 '22 20:07 kensukenk

Hmmm, for some reason there is still no visualization loading on my end.

kensukenk avatar Jul 11 '22 21:07 kensukenk

hi @kensukenk Can u show your Bokeh version? pip show bokeh

Also, any error in your cli when change the search filters in nuBoard?

kokseang-motional avatar Jul 12 '22 02:07 kokseang-motional

Name: bokeh Version: 2.4.3 Summary: Interactive plots and applications in the browser from Python Home-page: https://github.com/bokeh/bokeh Author: Bokeh Team Author-email: [email protected] License: BSD-3-Clause Location: /home/kensuke/anaconda3/envs/nuplan/lib/python3.9/site-packages Requires: PyYAML, Jinja2, numpy, tornado, packaging, typing-extensions, pillow Required-by: nuplan-devkit

And I get no errors in my cli when I change search filters. Only this when I run it: python nuplan/planning/script/run_nuboard.py simulation_path="[/home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03]" /home/kensuke/anaconda3/envs/nuplan/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. warnings.warn(

kensukenk avatar Jul 12 '22 02:07 kensukenk

Can u try press F12 to look at your browser console, any errors after you try searching sth?

kokseang-motional avatar Jul 12 '22 10:07 kokseang-motional

Hi @kensukenk Could u try to reinstall the devkit again? We have few updates

kokseang-motional avatar Jul 19 '22 01:07 kokseang-motional

Ah, so now it is a little bit different. The histogram tab still doesn't tell me anything, but now the scenarios tab displays text before I choose which scenario to pick. Afterward, it loads indefinitely it seems, but my terminal says

"(nuplan2) kensuke@Blade-15:~/nuplan-devkit$ python nuplan/planning/script/run_nuboard.py simulation_path="[/home/kensuke/nuplan/exp/exp/simulation_open_loop_boxes/2022.07.04.12.01.03]" /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/geopandas/_compat.py:112: UserWarning: The Shapely GEOS version (3.10.2-CAPI-1.16.0) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow. warnings.warn( INFO:nuplan.planning.script.builders.scenario_building_builder:Building AbstractScenarioBuilder... INFO:nuplan.database.nuplan_db.nuplandb_wrapper:Loaded maps DB INFO:nuplan.database.nuplan_db.nuplandb_wrapper:Loaded 72 log DBs
INFO:nuplan.planning.script.builders.scenario_building_builder:Building AbstractScenarioBuilder...DONE! INFO:nuplan.planning.nuboard.nuboard:Opening Bokeh application on http://localhost:5006/ INFO:bokeh.server.server:Starting Bokeh server version 2.4.3 (running on Tornado 6.1) INFO:bokeh.server.tornado:User authentication hooks NOT provided (default user enabled) INFO:tornado.access:200 GET / (127.0.0.1) 164.10ms INFO:tornado.access:200 GET /resource/style.css (127.0.0.1) 2.56ms INFO:tornado.access:200 GET /resource/spectre.min.css (127.0.0.1) 0.68ms INFO:tornado.access:200 GET /resource/spectre-exp.min.css (127.0.0.1) 0.33ms INFO:tornado.access:200 GET /resource/spectre-icons.min.css (127.0.0.1) 0.23ms INFO:tornado.access:200 GET /static/js/bokeh-gl.min.js?v=e5df31fd9010eacff0aa72d315264604b5e34972ba445acea6fce98080eecf33acf2d2986126360faaa5852813cffa16f6f6f4889923318300f062497c02da4e (127.0.0.1) 0.37ms INFO:tornado.access:200 GET /static/js/bokeh.min.js?v=3c61e952b808bb7e346ce828a565a5f23aaf7708d034fa9d0906403813355d45bb4e8d8b0b23a93f032c76831d4f0221846f28699c7f5147caa62e0d31668314 (127.0.0.1) 0.52ms INFO:tornado.access:200 GET /static/js/bokeh-mathjax.min.js?v=176c36fdbcd8fc1019fc828101a2804081a35baf4018d7f2633cd263156b593aa73112f400112b662daa0590138b74851bc91f1f2a5fbf5416ee8c876c3e0d0c (127.0.0.1) 1.33ms INFO:tornado.access:200 GET /static/js/bokeh-widgets.min.js?v=8a1ff6f5aa0d967f4998d275803bbb111d928fd9f605ef9e1f30cfd021df0e77224ee3d13f83edb3a942f6e4ccc569ee5dd8951a8aa6cb600602463b90c65a87 (127.0.0.1) 1.46ms INFO:tornado.access:200 GET /static/js/bokeh-tables.min.js?v=ae2903e57cf57f52819fdf4d938c648982b51c34f73b6e653a0f3bb3c8ab44f338505931ace43eafc1636e215492e2314acf54c54baffb47813b86b4923a7fe0 (127.0.0.1) 1.58ms INFO:tornado.access:200 GET /resource/scripts/utils.js (127.0.0.1) 0.44ms INFO:tornado.access:200 GET /resource/motional_logo.png (127.0.0.1) 1.10ms INFO:tornado.access:101 GET /ws (127.0.0.1) 0.60ms INFO:bokeh.server.views.ws:WebSocket connection opened INFO:bokeh.server.views.ws:ServerConnection created ERROR:bokeh.server.protocol_handler:error handling message message: Message 'PATCH-DOC' content: {'events': [{'kind': 'ModelChanged', 'model': {'id': '1021'}, 'attr': 'value', 'new': '71dfddf88e7e51e0'}], 'references': []} error: TypeError("a bytes-like object is required, not 'list'") Traceback (most recent call last): File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/server/protocol_handler.py", line 97, in handle work = await handler(message, connection) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/server/session.py", line 95, in _needs_document_lock_wrapper result = func(self, *args, **kwargs) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/server/session.py", line 289, in _handle_patch message.apply_to_document(self.document, self) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/protocol/messages/patch_doc.py", line 115, in apply_to_document invoke_with_curdoc(doc, lambda: doc.apply_json_patch(self.content, setter)) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/callbacks.py", line 408, in invoke_with_curdoc return f() File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/protocol/messages/patch_doc.py", line 115, in invoke_with_curdoc(doc, lambda: doc.apply_json_patch(self.content, setter)) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/document.py", line 391, in apply_json_patch DocumentPatchedEvent.handle_json(self, event_json, references, setter) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/events.py", line 259, in handle_json handler(doc, event_json, references, setter) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/events.py", line 467, in _handle_json patched_obj.set_from_json(attr, value, models=references, setter=setter) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/core/has_props.py", line 384, in set_from_json descriptor.set_from_json(self, json, models=models, setter=setter) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/core/property/descriptors.py", line 378, in set_from_json self._set(obj, old, value, setter=setter) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/core/property/descriptors.py", line 559, in _set self._trigger(obj, old, value, hint=hint, setter=setter) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/core/property/descriptors.py", line 637, in _trigger obj.trigger(self.name, old, value, hint, setter) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/model/model.py", line 567, in trigger super().trigger(descriptor.name, old, new, hint=hint, setter=setter) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/util/callback_manager.py", line 194, in trigger self.document.callbacks.notify_change(cast(Model, self), attr, old, new, hint, setter, invoke) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/callbacks.py", line 236, in notify_change self.trigger_on_change(event) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/callbacks.py", line 373, in trigger_on_change invoke_with_curdoc(doc, event.callback_invoker) File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/document/callbacks.py", line 408, in invoke_with_curdoc return f() File "/home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/util/callback_manager.py", line 191, in invoke callback(attr, old, new) File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/tabs/scenario_tab.py", line 375, in _scalar_scenario_name_select_on_change self._update_scenario_plot() File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/tabs/scenario_tab.py", line 312, in _update_scenario_plot self._simulation_plots = self._render_simulations() File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/tabs/scenario_tab.py", line 783, in _render_simulations self._simulation_figure_data = self.simulation_tile.render_simulation_tiles( File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/base/simulation_tile.py", line 266, in render_simulation_tiles self.init_simulations(figure_sizes=figure_sizes) File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/base/simulation_tile.py", line 225, in init_simulations simulation_figure = self._create_initial_figure(figure_index=figure_index, figure_sizes=figure_sizes) File "/home/kensuke/nuplan-devkit/nuplan/planning/nuboard/base/simulation_tile.py", line 192, in _create_initial_figure simulation_log = SimulationLog.load_data(simulation_file) File "/home/kensuke/nuplan-devkit/nuplan/planning/simulation/simulation_log.py", line 78, in load_data data = pickle.loads(data) TypeError: a bytes-like object is required, not 'list' /home/kensuke/anaconda3/envs/nuplan2/lib/python3.9/site-packages/bokeh/server/protocol_handler.py:101: RuntimeWarning: coroutine 'WSHandler.send_message' was never awaited work = connection.error(message, repr(e)) "

kensukenk avatar Jul 19 '22 01:07 kensukenk

Hi @kensukenk Have to rerun simulation since we updated the interface

kokseang-motional avatar Jul 21 '22 03:07 kokseang-motional

Close due to inactivity

patk-motional avatar Aug 23 '22 08:08 patk-motional

Full of various bugs.

927087638 avatar Oct 21 '23 15:10 927087638

hey, i met some issues, when i run nuplan_planner_tutorial.ipynb for the last cell "Launch nuboard(open in new tab), it seems stucked: image i have waited for more than 9 minutes as show above, and the nuboard shows below: image image can u tell me how to fix it? @patk-motional @kokseang-motional

jessapinkman avatar Nov 25 '23 05:11 jessapinkman