World-Models-Autonomous-Driving-Latest-Survey

A curated list of world model for autonmous driving. Keep updated.

Papers

• 2024-CarFormer: Self-Driving with Learned Object-Centric Representations ECCV 2024 Paper

• 2024-BEVWorld: A Multimodal World Model for Autonomous Driving via Unified BEV Latent Space arxiv Paper

• 2024-Planning with Adaptive World Models for Autonomous Driving arxiv; Planning; Paper

• 2024-UnO: Unsupervised Occupancy Fields for Perception and Forecasting Paper

• 2024-LAW: Enhancing End-to-End Autonomous Driving with Latent World Model Paper

• 2024-OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving Paper, Code

• 2024-Delphi: Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation Paper

• 2024-Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability from Shanghai AI Lab Paper

• 2024-DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving CVPR 2024; __Paper,

• 2024-UniPAD: A Universal Pre-training Paradigm for Autonomous Driving CVPR 2024; from Shanghai AI Lab Paper, Code

• 2024-GenAD: Generalized Predictive Model for Autonomous Driving CVPR 2024; from Shanghai AI Lab Paper

• 2024-Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving arxiv Paper

• 2024-ViDAR: Visual Point Cloud Forecasting enables Scalable Autonomous Driving CVPR 2024; Pre-training; from Shanghai AI Lab; NuScenes dataset Paper, Code

• 2024-Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion ICLR 2024; Future Prediction; from Waabi; NuScenes, KITTI Odemetry, Argoverse2 Lidar datasets Paper

• 2023-DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model arxiv; Generative AI Paper, Code

• 2023-MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations arxiv; Pre-training; CARLA dataset Paper

• 2023-Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving arxiv; Generative AI, Planning; NuScenes and Waymo datasets Paper

• 2023-ADriver-I: A General World Model for Autonomous Driving arxiv; Generative AI; NuScenes & one private dataset Paper

• 2023-OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving arxiv; Occupancy Future Prediction, Planning; Occ3D dataset for Occupancy Future Prediction, NuScenes for motion planning Paper, Code

• 2023-GAIA-1: A Generative World Model for Autonomous Driving arxiv; Generative AI; Wayve's private data Paper

  Related papers & tutorials to understand this paper:

  FDM for video diffusion decoder: Paper, Code

  Denoising diffusion tutorials: CVPR 2022 tutorial, class from UC Berkeley, Video

• 2023-DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving arxiv; Generative AI; NuScenes dataset Paper, Code (To be released soon)

• 2023-Neural World Models for Computer Vision 'PhD Thesis'; from Wayve Paper

• 2023-UniWorld: Autonomous Driving Pre-training via World Models arxiv; Pre-training; NuScenes dataset Paper

• 2022-Separating the World and Ego Models for Self-Driving ICLR 2022 workshop on Generalizable Policy Learning in the Physical World; from Yann Lecun's Group Paper, Code

• 2022-SEM2: Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model NeurIPS 2022 Deep Reinforcement Learning Workshop; RL; CARLA dataset Paper

• 2022-MILE: Model-Based Imitation Learning for Urban Driving NeurIPS 2022; RL; from Wayve Paper, Code

• 2022-Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models NeurIPS 2022 Paper, Code

• 2021-FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras ICCV 2019; Future Prediction; from Wayve; NuScenes, Lyft datasets Paper, Code

• 2021-Learning to drive from a world on rails CVPR 2021 Oral; RL Paper, Project Page, Code

• 2019-Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic ICLR 2019; Future Prediction; from Yann Lecun's Group Paper, Code

Workshops/Challenges

• 2024-1X World Model Challenge Challenges Link
• 2024-CVPR Workshop, Foundation Models for Autonomous Systems, Challenges, Track 4: Predictive World Model Challenges Link

Tutorials/Talks/

• 2023 from Wayve; Video
• 2022-Neural World Models for Autonomous Driving Video

Surveys that Contain World Models for AD

• 2024-World Models for Autonomous Driving: An Initial Survey arxiv Paper
• 2024-Data-Centric Evolution in Autonomous Driving: A Comprehensive Survey of Big Data System, Data Mining, and Closed-Loop Technologies arxiv Paper
• 2024-Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities arxiv Paper

Other General World Model Papers

• 2024-Hierarchical World Models as Visual Whole-Body Humanoid Controllers Paper
• 2024-Pandora: Towards General World Model with Natural Language Actions and Video States Paper
• 2024-Efficient World Models with Time-Aware and Context-Augmented Tokenization ICML 2024
• 2024-3D-VLA: A 3D Vision-Language-Action Generative World Model ICML 2024 Paper
• 2024-Newton from Archetype AI website Link
• 2024-MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators arxiv Paper, Code
• 2024-IWM: Learning and Leveraging World Models in Visual Representation Learning arxiv, from Yann Lecun's Group Paper
• 2024-Video as the New Language for Real-World Decision Making arxiv, Deepmind Paper
• 2024-Genie: Generative Interactive Environments Deepmind Paper, Website
• 2024-Sora OpenAI, Generative AI Link, Technical Report
• 2024-LWM: World Model on Million-Length Video And Language With RingAttention arxiv; Generative AI Paper, Code
• 2024-WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens arxiv; Generative AI Paper
• 2024-Video prediction models as rewards for reinforcement learning NeurIPS 2024 Paper, Code
• 2024-V-JEPA: Revisiting Feature Prediction for Learning Visual Representations from Video from Yann Lecun's Group Paper, Code
• 2023-Facing Off World Model Backbones: RNNs, Transformers, and S4 NeurIPS 2023 Paper
• 2023-I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture CVPR 2023; from Yann Lecun's Group Paper, Code
• 2023-Temporally Consistent Transformers for Video Generation ICML 2023 Paper, Code
• 2023-Learning to Model the World with Language arxiv Paper, Code
• 2023-Transformers are sample-efficient world models ICLR 2023;RL Paper, Code
• 2023-Gradient-based Planning with World Models arxiv; from Yann Lecun's Group; Planning; Paper
• 2023-World Models via Policy-Guided Trajectory Diffusion arxiv; RL; Paper
• 2023-DreamerV3: Mastering diverse domains through world models arxiv;RL; Paper, Code
• 2022-Daydreamer: World models for physical robot learning CoRL 2022; Robotics Paper, Code
• 2022-Masked World Models for Visual Control CoRL 2022; Robotics Paper, Code
• 2022-A Path Towards Autonomous Machine Intelligence openreview; from Yann Lecun's Group; General Roadmap for World Models; Paper; Slides1, Slides2, Slides3; Videos
• 2021-LEXA:Discovering and Achieving Goals via World Models NeurIPS 2021; Paper, Website & Code
• 2021-DreamerV2: Mastering Atari with Discrete World Models ICLR 2021; RL; from Google & Deepmind Paper, Code
• 2020-Dreamer: Dream to Control: Learning Behaviors by Latent Imagination ICLR 2020 Paper, Code
• 2019-Learning Latent Dynamics for Planning from Pixels ICML 2019 Paper, Code
• 2018-Model-Based Planning with Discrete and Continuous Actions arxiv; RL, Planning; from Yann Lecun's Group; Paper
• 2018-Recurrent world models facilitate policy evolution NeurIPS 2018; Paper, Code

Other Related Papers

• 2023-Occupancy Prediction-Guided Neural Planner for Autonomous Driving ITSC 2023; Planning, Neural Predicted-Guided Planning; Waymo Open Motion dataset Paper

Other Related Repos

Awesome-World-Model, Awesome-World-Models-for-AD , World models paper list from Shanghai AI lab, Awesome-Papers-World-Models-Autonomous-Driving.