Yunsong Zhou, Michael Simon, Zhenghao Peng, Sicheng Mo, Hongzi Zhu, Minyi Guo, and Bolei Zhou

• Presented by MetaDriverse, GenForce, and Shanghai Jiao Tong University
• :mailbox_with_mail: Primary contact: Yunsong Zhou ( [email protected] )
• arXiv paper | Blog TODO | Slides

Highlights

:fire: The first simulator-conditioned generative model for controllable driving scene generation with appearance and layout diversity.

:star2: SimGen addresses simulation to reality (Sim2Real) gaps via cascade diffusion paradigm, and follows layout guidance from simulators and cues of the rich text prompts to realistic driving scenarios.

:bar_chart: DIVA dataset comprises 147.5 hours of web videos and synthesized data for diverse scene generation and advancing Sim2Real research.

News

• [2024/06] SimGem paper released.
• [2024/06] DIVA dataset subset released.

Table of Contents

1. Highlights
2. News
3. TODO List
4. DIVA Dataset
5. License and Citation
6. Related Resources

TODO List

• [x] Release DIVA dataset
• [ ] Release SimGen code
• [ ] Toolkits for novel scene generation

DIVA Dataset

DIVA-Real. It collects driving videos from YouTube, covering a worldwide range of geography, weather, scenes, and traffic elements and preserving the appearance diversity of a wide range of traffic participants. Here we provide a sample of 🔗 YouTube video list we used. For privacy considerations, we are temporarily keeping the complete data labels private.

DIVA-Sim. The Sim2Real data is induced from the same real-world scenarios, in which we can obtain real-world map topology, layout, and raw sensor data. It also includes hazardous driving behaviors through interactions introduced by adversarial traffic generation. The digital twins (on nuScenes dataset) and safety-critical scenarios (on Waymo Open dataset) can be obtained through this 🔗data link.

License and Citation

All assets and code in this repository are under the Apache 2.0 license unless specified otherwise. The annotation data is under CC BY-NC-SA 4.0. Other datasets (including nuScenes, Waymo, and MetaDrive) inherit their own distribution licenses. Please consider citing our paper and project if they help your research.

@article{zhou2024simgen,
  title={SimGen: Simulator-conditioned Driving Scene Generation},
  author={Zhou, Yunsong and Simon, Michael and Peng, Zhenghao and Mo, Sicheng and Zhu, Hongzi and Guo, Minyi and Zhou, Bolei},
  journal={arXiv preprint arXiv:2406.09386},
  year={2024}
}

Related Resources

We acknowledge all the open-source contributors for the following projects to make this work possible:

• Uni-ControlNet | MetaDrive

You are welcome to follow other related work from , MetaDriverse, and GenForce.

• ELM | OpenScene | DriveAGI
• ScenarioNet | CAT | FreeControl