scenario_runner
scenario_runner copied to clipboard
carla-simulator
Scenario Generation Workflow
Thank you for making the CARLA Scenario Runner, it's a huge task and one that really completes the CARLA simulation.
We are utilizing it in a standard way: challenge an ADS with scenarios to improve and validate it.
We've developed our own map and are looking to decorate it with a library of scenarios. Our goal is to have a dense set of scenarios fill out a map. We've started this process are slowly generating scenarios for our map. I'm sure the pace will increase as we develop our skills, but our workflow seems very primitive and I'd like to hear what others are doing that might help us accelerate.
Currently our workflow is this:
- For each scenario we want to configure, create a new scenario configuration via the Scenario Runner's XML scheme. (As in, we're currently not using OpenSCENARIO).
- Using UE Editor, place a vehicle at a desired starting pose and change the scenario's ego_vehicle transform in the XML attributes.
- If an other_actor is present, move the vehicle to the desired pose and set that transform.
- Set any other configuration elements as necessary.
- Run the scenario using the scenario_runner.py to validate the scenario and tweak as needed.
We are repeating this process for each scenario, for each intersection and each direction at that intersection. We also have scenarios between intersections. This means that for each map, we have to spent quite a bit of time performing the scenario generation. We are looking for ways to improve this process to allow us to use the scenario runner to its fullest.
It appears that using OpenSCENARIO might be a solution to this, and we're willing to use it, but for us it shifts the problem toward developing a more verbose XML schema. To date we aren't aware of any good tools (GUI or otherwise) that have been developed to generate OpenSCENARIO data. However, we have found a list of tools that seem like a good start:
RControlStation: https://github.com/vedderb/rise_sdvp/tree/master/Linux/RControlStation
Eclipse OpenScenario Editor: described here: https://www.vinnova.se/globalassets/mikrosajter/ffi/dokument/slutrapporter-ffi/elektronik-mjukvara-och-kommunikation-rapporter/2016-05495eng.pdf but I couldn't find any real info about it...
VIRES: https://www.mscsoftware.com/product/virtual-test-drive
sim@openPASS: https://projects.eclipse.org/proposals/simopenpass
IPG CarMaker: https://ipg-automotive.com/products-services/simulation-software/carmaker-release-90/
These all seem like they're headed in the right direction, but frankly I'm concerned they aren't developed enough to be faster than just editing XML.
Related: https://github.com/esmini/esmini https://github.com/arauschert/OSC-ALKS-scenarios
So, to summarize, what are folks using to generate these scenarios? Is everyone just editing XML in a text editor similar to what we're doing and that's really the right approach? Or are there more developed approaches?
Related: #40 How do I get the exact location (x, y, z) on the map to spawn actors? #429 How to generate my own route and config files? #576 What tool do you use to create scenarios in the openscenario format? #636 Contributing with Scenarios (OpenScenario)
Thx @jimmyw404 for the nice summary.
Speaking of myself, I learned my way to work with the XMLs directly, but I also understand that this is very inconvenient. We are trying to bring up a more user friendly UI, maybe based on carlaviz. But we are in a very early stage, exploring the territory, so nothing to take for granted yet.
Regarding your listed options: CarMaker and VTD are unfortunately commercial solutions, hence they are not really an option here. As far as I know, there are also more close-source/private solutions available. However, in terms of public/free use, we are not aware of non-XML-based UI.
Thanks for the answer, @fabianoboril ! I look forward to seeing what your team comes up with. We're exploring the option of building a library of individual scenarios and then permuting them along a route to generate a large library of different challenges that can be selected and loaded in a deterministic way.
In other words, for a given intersection we might create a handful of individual FollowLeadingVehicle, SignalJunctionCrossingRoute, ControlLoss, DynamicObjectCrossing etc scenarios, and then we'll have a python script to generate different permutations that combine these into json scenario files that can be run in a RouteScenario. The key difference between this and the current Route Scenario process is that we want to run all scenarios in our scenario file.
As you can imagine, with a only a few base scenarios the number of permutations can get quite large, especially if we parameterize them (ex: different values for the dist_to_trigger in DynamicObjectCrossing), and many scenarios are mutually exclusive. Permuting a library of high-value scenarios that aren't just chaos will be a challenge, but I think it's worth pursuing. If your team has tried something like this I'd be curious to hear how it went!
If anything comes out of it I'll update this issue, and if you make any more progress on scenario generation or hear about similar efforts I'd love to know!
@jimmyw404 As you mentioned in the 2nd step , deciding the vehicle coordinates by positioning the actors in UE editor. Can you explain as how to load a required map present in carla to UE editor and position the actors . Thank you
@jimmyw404 As you mentioned in the 2nd step , deciding the vehicle coordinates by positioning the actors in UE editor. Can you explain as how to load a required map present in carla to UE editor and position the actors . Thank you
@tecena , I run "make launch" which launches the UE editor. I can then open the map I need.
I have two computers side by side, which makes it easy to use this workflow to position actors. The coordinate frame is the same though the UE editor's distances are in centimeters and the scenario (and rest of CARLA) is in meters.
https://carla.readthedocs.io/en/latest/build_windows/
AVL developed a Scenario Designer with a powerful GUI that fully supports OpenSCENARIO 1.0 and soon 1.1 and runs on Linux/Windows (disclaimer: I'm currently working for AVL)
Students and research institutions can use it for free via our University Partnership Program.
I already performed some tests with Carla scenario runner (0.9.11) and it worked out of the box. As you all know, Carla scenario runner is still a bit limited and does not support all OSC actions yet (docs) - and the OpenDRIVE standalone mode in Carla renders only grey roads. But I'm sure, the Carla community will overcome these issues :-)
i want to ask a simple question, how to write the xml file, i do not know its element meaning, when i go to the https://pyoscx.github.io/ page, i cant find a good tutorial
AVL developed a Scenario Designer with a powerful GUI that fully supports OpenSCENARIO 1.0 and soon 1.1 and runs on Linux/Windows (disclaimer: I'm currently working for AVL)
Students and research institutions can use it for free via our University Partnership Program.
I already performed some tests with Carla scenario runner (0.9.11) and it worked out of the box. As you all know, Carla scenario runner is still a bit limited and does not support all OSC actions yet (docs) - and the OpenDRIVE standalone mode in Carla renders only grey roads. But I'm sure, the Carla community will overcome these issues :-)
Visiting the site you provide Scenario Designer results in Page Not Found: Hello there! It looks like you‘ve reached a page that does not exist.
