Could this repo now support co-emulation with Apollov7

[yumianhuli2]

As the title says Thank you

[YuqiHuai]

Hi, do you have any source for more related information? I haven't heard anything related to this before.

[YuqiHuai]

I realized you meant "co-simulation". Yes, this project is intended to work for co-simulation and I have tried to use this with Apollo 7.

[GiantSeaweed]

I realized you meant "co-simulation". Yes, this project is intended to work for co-simulation and I have tried to use this with Apollo 7.

Hi. I am trying to use Python API to do "co-simulation" with Apollo r7.0.0 branch.

I am trying the code snippet at here. I disable the Perception/Camera/Traffic Lights modules. Did you encounter the situation where the localization module does not work? Here are some symptoms:

1. The python code will stuck on dv.setup_apollo(...). Specifically stuck at here.
2. I cannot enable Localization module on the Dreamview's Module Controller panel, either manually (it will immediately switch back to off) or via API.
3. The channel /apollo/localization/msf_status is always red in cyber_moniter. Channel /apollo/localization/pose is green.

Have you seen cases like this before? Did you have any additional changes on the Apollo r7.0.0 branch, or this repo, or python snippet?

Thank you!

[YuqiHuai]

Hi @GiantSeaweed yes I’ve seen similar problem. Are you using the right mode and the right vehicle on dreamview?

[GiantSeaweed]

I do not manually change anything on dreamview. They are all set by the python snippet. The mode is standard debug.

Here is the code:

I change the original dv.set_vehicle('Lincoln2017MKZ) to dv.set_vehicle('Lincoln2017MKZ_LGSVL') because apollo r7.0.0's dir is called so.

Am I using the wrong id in sim.add_agent()?

import os
import lgsvl
from environs import Env

env = Env()

sim = lgsvl.Simulator(
    env.str("LGSVL__SIMULATOR_HOST", lgsvl.wise.SimulatorSettings.simulator_host),
    env.int("LGSVL__SIMULATOR_PORT", lgsvl.wise.SimulatorSettings.simulator_port)
)

if sim.current_scene == lgsvl.wise.DefaultAssets.map_borregasave:
    sim.reset()
else:
    sim.load(lgsvl.wise.DefaultAssets.map_borregasave)

spawns = sim.get_spawn()

state = lgsvl.AgentState()
state.transform = spawns[0]

print("Adding agent")
ego = sim.add_agent(lgsvl.wise.DefaultAssets.ego_lincoln2017mkz_apollo5, lgsvl.AgentType.EGO, state)
ego.connect_bridge(
    env.str("LGSVL__AUTOPILOT_0_HOST", lgsvl.wise.SimulatorSettings.bridge_host),
    env.int("LGSVL__AUTOPILOT_0_PORT", lgsvl.wise.SimulatorSettings.bridge_port)
)

# Dreamview setup
print('Connecting to dv')
dv = lgsvl.dreamview.Connection(sim, ego, env.str("LGSVL__AUTOPILOT_0_HOST", "127.0.0.1"))
dv.set_hd_map('Borregas Ave')
dv.set_vehicle('Lincoln2017MKZ_LGSVL')
modules = [
    'Localization',
    # 'Perception',
    'Transform',
    'Routing',
    'Prediction',
    'Planning',
    # 'Camera',
    # 'Traffic Light',
    'Control'
]
destination = spawns[0].destinations[0]

print('DV setting up apollo')
dv.setup_apollo(destination.position.x, destination.position.z, modules)

print('Sim start running')
sim.run()

[YuqiHuai]

@GiantSeaweed I belive you have to call https://github.com/lgsvl/PythonAPI/blob/9bc29c9aee5e98d4d99e6a410c04b153b8f7feef/lgsvl/dreamview/dreamview.py#L204 to change mode to "Mkz Lgsvl". The default is "Mkz Standard Debug" which always fails for me.

You can try adding dv.set_setup_mode('Mkz Lgsvl') right after you set_vehicle to see if it works,

Also for set up vehicle I don't think there is a _ between MKZ and LGSVL. You can take a look at Dreamview's interface to see what are the options.

[GiantSeaweed]

Hi @YuqiHuai

I set the mode to Mkz Lgsvl and all modules can be enabled on dreamview!

About the _, I think it doesn't matter (you can see the processing here.

But I got some new problems:

1. The code is still stuck at here. Only if I manually click the 'Play' button in lgsvl simulator that the code will go on.

2. Also the car will not move forward even if all modules are enabled. Seems there is a huge delay.

3. Sometimes I see the following error message in Dreamview Console. Does it matter ?

- Failed to load default routing. Please make sure the file exists at /apollo/modules/map/data/borregas_ave_default_cycle_routing.txt17:23:48
- Failed to load default POI. Please make sure the file exists at /apollo/modules/map/data/borregas_ave/default_end_way_point.txt

Thank you very much!

[YuqiHuai]

I’m reopening this issue so I can track and work on resolving it.

[YuqiHuai]

@GiantSeaweed sorry for the late reply. I have tested the following code and I am able to get it working with Apollo 7.0 but not 8.0. I am not sure about the root cause yet.

import os
import lgsvl
from environs import Env

env = Env()

sim = lgsvl.Simulator(
    env.str("LGSVL__SIMULATOR_HOST", lgsvl.wise.SimulatorSettings.simulator_host),
    env.int("LGSVL__SIMULATOR_PORT", lgsvl.wise.SimulatorSettings.simulator_port)
)

if sim.current_scene == lgsvl.wise.DefaultAssets.map_borregasave:
    sim.reset()
else:
    sim.load(lgsvl.wise.DefaultAssets.map_borregasave)

spawns = sim.get_spawn()

state = lgsvl.AgentState()
state.transform = spawns[0]

ego = sim.add_agent('2e966a70-4a19-44b5-a5e7-64e00a7bc5de', lgsvl.AgentType.EGO, state)
# source: https://github.com/MingfeiCheng/AV-Fuzzer/blob/aa0f96c35088502189f7a9343d9ec7b0cee46b55/simulation/simulator.py#L12
ego.connect_bridge(
    env.str("LGSVL__AUTOPILOT_0_HOST", lgsvl.wise.SimulatorSettings.bridge_host),
    env.int("LGSVL__AUTOPILOT_0_PORT", lgsvl.wise.SimulatorSettings.bridge_port)
)

# Dreamview setup
dv = lgsvl.dreamview.Connection(sim, ego, env.str("LGSVL__AUTOPILOT_0_HOST", "127.0.0.1"))
dv.set_hd_map('Borregas Ave')
dv.set_vehicle('Lincoln2017MKZ')
modules = [
    'Localization',
    'Perception',
    'Transform',
    'Routing',
    'Prediction',
    'Planning',
    'Traffic Light',
    'Control'
]
destination = spawns[0].destinations[0]
dv.setup_apollo(destination.position.x, destination.position.z, modules)

sim.run(30)

[GiantSeaweed]

Hi @YuqiHuai . Thanks for your reply! It works with your script!!

I found the only difference is the id in sim_add_agent(). I didn't realize the sensor setting can have such effect.

Btw, it seems that dreamview will remember some settings. If I succeed running via your id, then I can have another execution by my original id. (But the reversed order doesn't work),

Also, I still need to click the Play button of SVL (just for the first time after launching). Is that also the case for you?

[YuqiHuai]

Hi @GiantSeaweed . No worries! Glad it helps.

Right, I am using the ID provided by Mingfei, who implemented a working version of AV-FUZZER (https://github.com/MingfeiCheng/AV-Fuzzer/blob/aa0f96c35088502189f7a9343d9ec7b0cee46b55/simulation/simulator.py#L12) and I have been using his configuration for a while.

What you are observing is so weird, I don't have an explanation as to why this might be happening. Maybe you can try to compare sensor configuration of the different IDs.

I also noticed I needed to click the Play button of SVL, and I don't remember needing to press that before. So yes, it is also the case for me. Maybe I should spend some time figuring out what's going on about that as well...

[GiantSeaweed]

Thanks for your help. Yeah. But just need to click for the first time.:)

[YuqiHuai]

@GiantSeaweed, You are welcome! Yes, it is the same for me as well, only the first time.

[ceresshadows]

@GiantSeaweed sorry for the late reply. I have tested the following code and I am able to get it working with Apollo 7.0 but not 8.0. I am not sure about the root cause yet.

import os
import lgsvl
from environs import Env

env = Env()

sim = lgsvl.Simulator(
    env.str("LGSVL__SIMULATOR_HOST", lgsvl.wise.SimulatorSettings.simulator_host),
    env.int("LGSVL__SIMULATOR_PORT", lgsvl.wise.SimulatorSettings.simulator_port)
)

if sim.current_scene == lgsvl.wise.DefaultAssets.map_borregasave:
    sim.reset()
else:
    sim.load(lgsvl.wise.DefaultAssets.map_borregasave)

spawns = sim.get_spawn()

state = lgsvl.AgentState()
state.transform = spawns[0]

ego = sim.add_agent('2e966a70-4a19-44b5-a5e7-64e00a7bc5de', lgsvl.AgentType.EGO, state)
# source: https://github.com/MingfeiCheng/AV-Fuzzer/blob/aa0f96c35088502189f7a9343d9ec7b0cee46b55/simulation/simulator.py#L12
ego.connect_bridge(
    env.str("LGSVL__AUTOPILOT_0_HOST", lgsvl.wise.SimulatorSettings.bridge_host),
    env.int("LGSVL__AUTOPILOT_0_PORT", lgsvl.wise.SimulatorSettings.bridge_port)
)

# Dreamview setup
dv = lgsvl.dreamview.Connection(sim, ego, env.str("LGSVL__AUTOPILOT_0_HOST", "127.0.0.1"))
dv.set_hd_map('Borregas Ave')
dv.set_vehicle('Lincoln2017MKZ')
modules = [
    'Localization',
    'Perception',
    'Transform',
    'Routing',
    'Prediction',
    'Planning',
    'Traffic Light',
    'Control'
]
destination = spawns[0].destinations[0]
dv.setup_apollo(destination.position.x, destination.position.z, modules)

sim.run(30)

Hi, YuqiHuai

May I ask where should I edit in this code if I want to run Apollo on another computer with LGSVL? I've tried only to change the ip address from 172.0.0.1 to the Apollo computer's ip, but the simulator said disconnected (screenshot as follow).

I used to succeed in co-simulating on two computers but with Apollo 6 (code as follow, is edited from Python-API/examples/SampleTestCases/random-traffic-local.py). Do you have any ideas about this problem?

Thank you so much!

#!/usr/bin/env python3
#
# Copyright (c) 2020 LG Electronics, Inc.
#
# This software contains code licensed as described in LICENSE.
#

from datetime import datetime
from environs import Env
import random
import lgsvl


'''
LGSVL__AUTOPILOT_0_HOST             IP address of the computer running the bridge to connect to
LGSVL__AUTOPILOT_0_PORT             Port that the bridge listens on for messages
LGSVL__AUTOPILOT_0_VEHICLE_CONFIG   Vehicle configuration to be loaded in Dreamview (Capitalization and spacing must match the dropdown in Dreamview)
LGSVL__AUTOPILOT_HD_MAP             HD map to be loaded in Dreamview (Capitalization and spacing must match the dropdown in Dreamview)
LGSVL__MAP                          ID of map to be loaded in Simulator
LGSVL__RANDOM_SEED                  Simulation random seed
LGSVL__SIMULATION_DURATION_SECS     How long to run the simulation for
LGSVL__SIMULATOR_HOST               IP address of computer running simulator (Master node if a cluster)
LGSVL__SIMULATOR_PORT               Port that the simulator allows websocket connections over
LGSVL__VEHICLE_0                    ID of EGO vehicle to be loaded in Simulator
'''

env = Env()

SIMULATOR_HOST = env.str("LGSVL__SIMULATOR_HOST", "127.0.0.1")
SIMULATOR_PORT = env.int("LGSVL__SIMULATOR_PORT", 8181)
# BRIDGE_HOST = env.str("LGSVL__AUTOPILOT_0_HOST", "127.0.0.1")
# BRIDGE_PORT = env.int("LGSVL__AUTOPILOT_0_PORT", 9090)

BRIDGE_HOST = '192.168.31.31'
BRIDGE_PORT = 9090

LGSVL__AUTOPILOT_HD_MAP = env.str("LGSVL__AUTOPILOT_HD_MAP", "san_francisco")
LGSVL__AUTOPILOT_0_VEHICLE_CONFIG = env.str("LGSVL__AUTOPILOT_0_VEHICLE_CONFIG", 'Lincoln2017MKZ')
LGSVL__SIMULATION_DURATION_SECS = 120.0
LGSVL__RANDOM_SEED = env.int("LGSVL__RANDOM_SEED", 51472)

vehicle_conf = env.str("LGSVL__VEHICLE_0", lgsvl.wise.DefaultAssets.ego_lincoln2017mkz_apollo5_modular)
scene_name = env.str("LGSVL__MAP", "12da60a7-2fc9-474d-a62a-5cc08cb97fe8")
sim = lgsvl.Simulator(SIMULATOR_HOST, SIMULATOR_PORT)
try:
    print("Loading map {}...".format(scene_name))
    sim.load(scene_name, LGSVL__RANDOM_SEED) # laod map with random seed
except Exception:
    if sim.current_scene == scene_name:
        sim.reset()
    else:
        sim.load(scene_name)


# reset time of the day
sim.set_date_time(datetime(2020, 7, 1, 15, 0, 0, 0), True)

spawns = sim.get_spawn()
# select spawn deterministically depending on the seed
spawn_index = LGSVL__RANDOM_SEED % len(spawns)

state = lgsvl.AgentState()
state.transform = spawns[spawn_index]  # TODO some sort of Env Variable so that user/wise can select from list
print("Loading vehicle {}...".format(vehicle_conf))
ego = sim.add_agent("2e966a70-4a19-44b5-a5e7-64e00a7bc5de", 
    lgsvl.AgentType.EGO, state)

print("Connecting to bridge...")
# The EGO is now looking for a bridge at the specified IP and port
ego.connect_bridge(BRIDGE_HOST, BRIDGE_PORT)
print("bridge connected")
def on_collision(agent1, agent2, contact):
    raise Exception("{} collided with {}".format(agent1, agent2))
    sys.exit()

ego.on_collision(on_collision)

dv = lgsvl.dreamview.Connection(sim, ego, BRIDGE_HOST)
dv.set_hd_map(LGSVL__AUTOPILOT_HD_MAP)
dv.set_vehicle(LGSVL__AUTOPILOT_0_VEHICLE_CONFIG)

destination_index = LGSVL__RANDOM_SEED % len(spawns[spawn_index].destinations)
destination = spawns[spawn_index].destinations[destination_index] # TODO some sort of Env Variable so that user/wise can select from list

default_modules = [
    'Localization',
    'Transform',
    'Routing',
    'Prediction',
    'Planning',
    'Control',
    #'Recorder'
]

dv.disable_apollo()
dv.setup_apollo(destination.position.x, destination.position.z, default_modules)

print("adding npcs")
sim.add_random_agents(lgsvl.AgentType.NPC)
sim.add_random_agents(lgsvl.AgentType.PEDESTRIAN)

sim.run(LGSVL__SIMULATION_DURATION_SECS)

[YuqiHuai]

@yellinshadows hi. Sorry I don’t have 2 machines to test this out. But temporarily maybe you can try using a previous version of SVL (2020.06). I talked with another person and he mentioned that version did not depend on having the cloud so maybe that is the simplest solution unless you have to use 2021.03.

[JasonBourne1998]

Hi @YuqiHuai, I have tried your script and it can be run on Apollo 7.0. However, the car you use is '2e966a70-4a19-44b5-a5e7-64e00a7bc5de', this car uses 3D ground Truth, which means it directly transfers the simulator's detecting data(NPC, Ped) to the Apollo. But in a real-world case, Apollo should collect the camera/Lidar data first and analysis it with a perception module like what Apollo 5.0 do on LGSVL (start from 31:43 https://www.youtube.com/watch?v=Ucr0aM334_k). On apollo 6.0, seems there is a problem with camera, so lgsvl uses module-testing configuration such as 2e966a70-4a19-44b5-a5e7-64e00a7bc5de to avoid this problem. But what if Apollo 7.0 has fixed camera's problem, how could we use the appropriate configuration to make Apollo can run full-analysis on lgsvl (I have tried the original Apollo 5.0's configuration '47b529db-0593-4908-b3e7-4b24a32a0f70' but seems not working )? Do you have some advice? Thanks in advance!

[YuqiHuai]

Hi @JasonBourne1998 I agree with you that eventually we should be testing Apollo E2E (i.e., not relying on 3D ground truth anymore), but at the moment I did not find enough evidence of SVL actually working with Apollo's perception module.

The first time I started using SVL was when I discovered ADFuzz (might also be known as AutoFuzz), and that project is suggesting the ID (2e966a...) you provided here. When I looked at the collision which they submitted to Apollo, I saw a significant flickering problem in obstacle detection. The paper discussed finding 76 collisions, but when I tried to run experiments with ADFuzz, collision only occurred when obstacle detection flickers and it rarely occurred when using 3D ground truth.

I decided to investigate whether this flickering problem is Apollo's fault, but I saw posts like SVL Issue 1719 and Apollo Issue 14072. These issues make me believe SVL isn't working with Apollo's perception, but no one knows if this is an Apollo problem or an SVL problem.

Even in the YouTube video you shared, at 36:52, many cars are in the junction, but Apollo seems to have detected none. It is very hard for me to believe Apollo's perception module is this bad. A few seconds later in the video, obstacles are detected, but I did not see what was changed between the 2 simulations.

With SVL sunsetted, I cannot figure out the root cause of the perception issue. Before anyone has any concrete evidence of this perception issue being solely Apollo's responsibility, I actually suggest relying on 3D ground truth to test the prediction module and the planning module. Everything I said here is also why I choose to rely on 3D ground truth for now and keep suggesting vehicle ID 2e96a70... which came from here.

[JasonBourne1998]

Hi @JasonBourne1998 I agree with you that eventually we should be testing Apollo E2E (i.e., not relying on 3D ground truth anymore), but at the moment I did not find enough evidence of SVL actually working with Apollo's perception module.

The first time I started using SVL was when I discovered ADFuzz (might also be known as AutoFuzz), and that project is suggesting the ID (2e966a...) you provided here. When I looked at the collision which they submitted to Apollo, I saw a significant flickering problem in obstacle detection. The paper discussed finding 76 collisions, but when I tried to run experiments with ADFuzz, collision only occurred when obstacle detection flickers and it rarely occurred when using 3D ground truth.

I decided to investigate whether this flickering problem is Apollo's fault, but I saw posts like SVL Issue 1719 and Apollo Issue 14072. These issues make me believe SVL isn't working with Apollo's perception, but no one knows if this is an Apollo problem or an SVL problem.

Even in the YouTube video you shared, at 36:52, many cars are in the junction, but Apollo seems to have detected none. It is very hard for me to believe Apollo's perception module is this bad. A few seconds later in the video, obstacles are detected, but I did not see what was changed between the 2 simulations.

With SVL sunsetted, I cannot figure out the root cause of the perception issue. Before anyone has any concrete evidence of this perception issue being solely Apollo's responsibility, I actually suggest relying on 3D ground truth to test the prediction module and the planning module. Everything I said here is also why I choose to rely on 3D ground truth for now and keep suggesting vehicle ID 2e96a70... which came from here.

I have got your opinion, thanks！ BTW, have you tried carla's Apollo 7.0 (https://github.com/guardstrikelab/carla_apollo_bridge) ? Carla can support full-analysis of Apollo 7.0, but I thinks Carla's problem is that its inner map is too simple, for ADS testing, I think San_Francsico in LGSVL is a good map. Since SVL sunsetted, I think the most feasible way is to create some complex maps in carla ?

[YuqiHuai]

@JasonBourne1998 I am aware of this Carla-Apollo bridge but I haven't had the time to try it yet. San_Francisco is definitely a good map! The work that I focus on relies on using Apollo's Sim-Control so I can just work with any HD map in the binary format (borregas_ave, san_mateo, sunnyvale_loop, etc.). But I will definitely check out Carla sometime soon!