toolkit
toolkit copied to clipboard
mapbench
Is Your HD Map Constructor Reliable under Sensor Corruptions?
English | 简体中文
Is Your HD Map Constructor Reliable under Sensor Corruptions?
Xiaoshuai Hao1 Mengchuan Wei1 Yifan Yang1 Haimei Zhao Hui Zhang1 Yi Zhou1 Qiang Wang1 Weiming Li Lingdong Kong3,‡ Jing Zhang2,‡ 1Samsung R&D Institute China-Beijing 2The University of Sydney 3National University of Singapore
About
MapBench is the first comprehensive benchmark designed to evaluate the out-of-domain robustness of HD map construction methods against various sensor corruptions.
 |
|---|
| Our benchmark encompasses a total of 16 corruption types for HD map construction, which can be categorized into exterior, interior, and sensor failure scenarios. Besides, we define 13 multi-sensor corruptions by combining the camera and LiDAR sensor failure types. |
Updates
- [2024.06] - Launch of the
MapBenchbenchmark. In this version, we include a total of 31 HD map construction models, evaluated on 29 different camera and LiDAR corruption types across 3 severity levels.
Outline
- Benchmark Definition
- Installation
- Data Preparation
- Getting Started
- Model Zoo
- Benchmark
- TODO List
- Citation
- License
- Acknowledgements
:bar_chart: Benchmark Definition
MapBench consists of a total of 29 different sensor corruption scenarios, including 8 types of camera corruptions, 8 types of LiDAR corruptions, and 13 types of camera-LiDAR corruption combinations.
LiDAR Sensor Corruptions
 |
 |
 |
 |
| Wet Ground | Snow | Beam Missing | Incomplete Echo |
 |
 |
 |
 |
| Fog | Motion Blur | Crosstalk | Cross-Sensor |
| Type | Description | Parameter | Easy | Moderate | Hard |
|---|---|---|---|---|---|
| Wet Ground | significantly attenuated laser echoes | (water height, noise floor) | (0.2, 0.2) | (1.0, 0.3) | (1.2, 0.7) |
| Snow | back-scattering and attenuation of LiDAR points | (snowfall rate, terminal velocity) | (0.5, 2.0) | (1.0, 1.6) | (2.5, 1.6) |
| Beam Missing | loss of certain light impulses | beam number to drop | 8 | 16 | 24 |
| Incomplete Echo | incomplete LiDAR readings | drop ratio | 0.75 | 0.85 | 0.95 |
| Fog | back-scattering and attenuation of LiDAR points | beta | 0.008 | 0.05 | 0.2 |
| Motion Blur | blur caused by vehicle movement | trans std | 0.2 | 0.3 | 0.4 |
| Crosstalk | light impulses interference | percentage | 0.03 | 0.07 | 0.12 |
| Cross-Sensor | cross sensor data | beam number to drop | 8 | 16 | 20 |
Camera Sensor Corruptions
 |
 |
 |
| Brightness | Low-Light | Fog |
 |
 |
 |
| Snow | Motion Blur | Color Quant |
| Type | Description | Parameter | Easy | Moderate | Hard |
|---|---|---|---|---|---|
| Brightness | varying daylight intensity | adjustment in HSV space | 0.2 | 0.4 | 0.5 |
| Low-Light | varying daylight intensity | scale factor | 0.5 | 0.4 | 0.3 |
| Fog | a visually obstructive form of precipitation | (thickness, smoothness) | (2.0, 2.0) | (2.5, 1.5) | (3.0, 1.4) |
| Snow | a visually obstructive form of precipitation | (mean, std, scale, threshold, blur radius, blur std, blending ratio) | (0.1, 0.3, 3.0, 0.5, 10.0, 4.0, 0.8) | (0.2, 0.3, 2, 0.5, 12, 4, 0.7) | (0.55, 0.3, 4, 0.9, 12, 8, 0.7) |
| Motion Blur | moving camera quickly | (radius, sigma) | (15, 5) | (15, 12) | (20, 15) |
| Color Quant | reducing the number of colors | bit number | 5 | 4 | 3 |
Camera Sensor Failures
 |
 |
| Frame Lost | Camera Crash |
| Type | Description | Parameter | Easy | Moderate | Hard |
|---|---|---|---|---|---|
| Frame Lost | dropping temporal frames | probability of frame dropping | 2/6 | 4/6 | 5/6 |
| Camera Crash | dropping view images | number of dropped cameras | 2 | 4 | 5 |
:gear: Installation
For details related to installation, kindly refer to INSTALL.md.
:hotsprings: Data Preparation
Our datasets are hosted by OpenDataLab.
OpenDataLab is a pioneering open data platform for the large AI model era, making datasets accessible. By using OpenDataLab, researchers can obtain free formatted datasets in various fields.
Kindly refer to DATA_PREPARE.md for the details to prepare the training and evaluation datasets.
:rocket: Getting Started
To learn more usage about this codebase, kindly refer to GET_STARTED.md.
:high_brightness: Model Zoo
 HD Map Construction
:golf: Benchmark
The mean average precision (mAP) is consistently used as the main indicator for evaluating model performance in our HD Map construction benchmark.
The following two metrics are adopted to compare among models' robustness under sensor corruptions:
- mCE (the lower the better): The average corruption error (in percentage) of a candidate model compared to the baseline model, which is calculated among all corruption types across three severity levels.
- mRR (the higher the better): The average resilience rate (in percentage) of a candidate model compared to its "clean" performance, which is calculated among all corruption types across three severity levels.
:red_car: Camera-Only Benchmarking Results
 |
|---|
| Model | mCE | mRR | Clean | Camera | Frame | Quant | Motion | Bright | Dark | Fog | Snow |
|---|---|---|---|---|---|---|---|---|---|---|---|
| HDMapNet | 18.78 | 43.3 | 23.0 | 4.6 | 5.1 | 18.9 | 20.8 | 16.7 | 9.3 | 10.6 | 5.2 |
| VectorMapNet | 148.5 | 40.6 | 40.9 | 13.9 | 12.3 | 26.6 | 29.7 | 25.2 | 7.8 | 18.3 | 2.9 |
| PivotNet | 96.3 | 45.2 | 57.4 | 17.1 | 16.7 | 36.4 | 34.1 | 43.5 | 16.5 | 37.4 | 4.6 |
| BeMapNet | 78.5 | 50.3 | 59.8 | 18.8 | 18.5 | 38.1 | 35.3 | 50.7 | 23.2 | 46.5 | 9.6 |
| MapTR | 100.0 | 49.3 | 50.3 | 15.0 | 14.2 | 35.4 | 23.5 | 44.3 | 22.7 | 38.5 | 3.8 |
| MapTRv2 | 72.6 | 51.4 | 61.5 | 18.8 | 18.2 | 45.3 | 31.0 | 54.9 | 32.3 | 50.7 | 1.1 |
| StreamMapNet | 64.8 | 54.4 | 63.4 | 13.4 | 15.5 | 48.1 | 44.3 | 57.0 | 36.1 | 52.4 | 9.1 |
| HIMap | 56.9 | 56.6 | 65.5 | 19.4 | 19.0 | 52.0 | 42.5 | 60.9 | 40.6 | 57.1 | 5.1 |
:blue_car: LiDAR-Only Benchmarking Results
 |
|---|
| Model | mCE | mRR | Clean | Fog | Wet | Snow | Motion | Beam | Crosstalk | Echo | Sensor |
|---|---|---|---|---|---|---|---|---|---|---|---|
| VectorMapNet | 94.9 | 63.4 | 34.0 | 15.7 | 20.3 | 15.9 | 28.8 | 19.2 | 19.7 | 31.3 | 9.5 |
| MapTR | 100.0 | 55.1 | 55.6 | 19.9 | 19.1 | 9.6 | 27.1 | 16.5 | 16.3 | 32.3 | 6.4 |
| MapTRv2 | 74.6 | 57.2 | 61.5 | 28.5 | 29.5 | 10.3 | 36.9 | 27.9 | 15.4 | 44.7 | 14.0 |
| HIMap | 73.1 | 59.2 | 64.3 | 26.6 | 24.6 | 16.1 | 37.4 | 24.4 | 26.7 | 43.1 | 10.8 |
:taxi: Camera-LiDAR Fusion Benchmarking Results
 |
|---|
| Model | Modality | Camera | Lidar | APped | APdiv | APbou | mAP |
|---|---|---|---|---|---|---|---|
| MapTR | C & L | ✓ | ✓ | 55.9 | 62.3 | 69.3 | 62.5 |
| MapTR | C | ✓ | — | 46.3 | 51.5 | 53.1 | 50.3 |
| MapTR | C | Camera Crash | — | 18.0 | 14.5 | 12.4 | 15.0 |
| MapTR | C | Frame Lost | — | 13.9 | 15.1 | 13.4 | 14.2 |
| MapTR | C & L | ✗ | ✓ | 15.0 | 18.2 | 34.4 | 22.5 |
| MapTR | C & L | Camera Crash | ✓ | 32.5 | 36.5 | 48.4 | 39.1 |
| MapTR | C & L | Frame Lost | ✓ | 29.1 | 33.7 | 46.1 | 36.3 |
| MapTR | L | — | ✓ | 26.6 | 31.7 | 41.8 | 33.4 |
| MapTR | L | — | Incomplete Echo | 26.3 | 29.9 | 40.6 | 32.3 |
| MapTR | L | — | Crosstalk | 13.6 | 15.0 | 20.3 | 16.3 |
| MapTR | L | — | Cross-Sensor | 3.5 | 6.6 | 8.9 | 6.4 |
| MapTR | C & L | ✓ | ✗ | 20.7 | 27.4 | 13.1 | 20.4 |
| MapTR | C & L | ✓ | Incomplete Echo | 47.9 | 55.2 | 62.2 | 55.1 |
| MapTR | C & L | ✓ | Crosstalk | 36.7 | 42.5 | 45.3 | 41.5 |
| MapTR | C & L | ✓ | Cross-Sensor | 33.9 | 42.9 | 42.0 | 39.6 |
| MapTR | C & L | Camera Crash | Incomplete Echo | 32.4 | 35.6 | 47.8 | 38.6 |
| MapTR | C & L | Camera Crash | Crosstalk | 19.7 | 21.6 | 26.9 | 22.7 |
| MapTR | C & L | Camera Crash | Cross-Sensor | 18.4 | 20.8 | 23.2 | 20.8 |
| MapTR | C & L | Frame Lost | Incomplete Echo | 28.9 | 32.8 | 45.5 | 35.8 |
| MapTR | C & L | Frame Lost | Crosstalk | 16.9 | 19.9 | 25.5 | 20.8 |
| MapTR | C & L | Frame Lost | Cross-Sensor | 15.8 | 19.4 | 22.2 | 19.1 |
| Model | Modality | Camera | Lidar | APped | APdiv | APbou | mAP |
|---|---|---|---|---|---|---|---|
| HIMap | C & L | ✓ | ✓ | 71.0 | 72.4 | 79.4 | 74.3 |
| HIMap | C | ✓ | — | 62.2 | 66.5 | 67.9 | 65.5 |
| HIMap | C | Camera Crash | — | 27.3 | 19.4 | 11.6 | 19.4 |
| HIMap | C | Frame Lost | — | 21.7 | 19.1 | 16.1 | 19.0 |
| HIMap | C & L | ✗ | ✓ | 40.9 | 46.4 | 74.7 | 50.7 |
| HIMap | C & L | Camera Crash | ✓ | 36.3 | 27.7 | 20.9 | 28.3 |
| HIMap | C & L | Frame Lost | ✓ | 29.9 | 25.0 | 23.8 | 26.2 |
| HIMap | L | — | ✓ | 54.8 | 64.7 | 73.5 | 64.3 |
| HIMap | L | — | Incomplete Echo | 35.4 | 41.1 | 52.7 | 43.1 |
| HIMap | L | — | Crosstalk | 20.9 | 23.8 | 35.3 | 26.7 |
| HIMap | L | — | Cross-Sensor | 7.8 | 10.2 | 14.4 | 10.8 |
| HIMap | C & L | ✓ | ✗ | 30.7 | 38.7 | 29.0 | 32.8 |
| HIMap | C & L | ✓ | Incomplete Echo | 59.1 | 63.7 | 69.9 | 64.2 |
| HIMap | C & L | ✓ | Crosstalk | 54.1 | 57.5 | 63.4 | 58.3 |
| HIMap | C & L | ✓ | Cross-Sensor | 44.2 | 50.7 | 50.8 | 48.5 |
| HIMap | C & L | Camera Crash | Incomplete Echo | 36.2 | 26.9 | 20.5 | 27.9 |
| HIMap | C & L | Camera Crash | Crosstalk | 29.2 | 19.3 | 12.9 | 20.5 |
| HIMap | C & L | Camera Crash | Cross-Sensor | 23.1 | 13.8 | 5.9 | 14.3 |
| HIMap | C & L | Frame Lost | Incomplete Echo | 29.9 | 24.4 | 23.5 | 25.9 |
| HIMap | C & L | Frame Lost | Crosstalk | 23.6 | 18.9 | 18.0 | 20.2 |
| HIMap | C & L | Frame Lost | Cross-Sensor | 17.7 | 14.3 | 11.2 | 14.4 |
:memo: TODO List
- [x] Initial release. 🚀
- [x] Add scripts for creating common corruptions.
- [ ] Add evaluation scripts on corruption sets.
- [ ] ...
Citation
If you find this work helpful, please kindly consider citing our paper:
@article{hao2024mapbench,
author = {Xiaoshuai Hao and Mengchuan Wei and Yifan Yang and Haimei Zhao and Hui Zhang and Yi Zhou and Qiang Wang and Weiming Li and Lingdong Kong and Jing Zhang},
title = {Is Your HD Map Constructor Reliable under Sensor Corruptions?},
journal={arXiv preprint arXiv:2406.12214},
year = {2024},
}
License
This work is under the Apache License Version 2.0, while some specific operations in this codebase might be with other licenses. Please refer to LICENSE.md for a more careful check, if you are using our code for commercial matters.
mapbench