autoware
autoware copied to clipboard
autowarefoundation
Create a rosbag anonymizer tool
Checklist
- [X] I've read the contribution guidelines.
- [X] I've searched other issues and no duplicate issues were found.
- [X] I've agreed with the maintainers that I can plan this task.
Description
The Autoware Foundation seeks to develop a tool that anonymizes camera data within rosbags, specifically targeting the blurring of faces and license plates to maintain privacy. This initiative aims to enable the secure sharing of rosbags containing camera data amongst member companies and the wider community.
Purpose
The primary goal is to ensure the privacy of individuals captured in camera data shared within the Autoware ecosystem. By creating a tool that can anonymize sensitive information in rosbags, we facilitate a safer, privacy-compliant exchange of data that can be used for research, development, and testing of autonomous vehicle technologies.
Possible approaches
- Fork the autodistill project to the Autoware Foundation's repository, as it is under the Apache 2.0 license and could serve as a starting point.
- Manually test the SAM model for its effectiveness in anonymizing faces and license plates within camera data.
- Based on the results, develop a tool (either as a standalone application or integrated within the autodistill project) that inputs a rosbag and outputs an anonymized version of the same data, with faces and license plates blurred.
Definition of done
- [X] The autodistill project is successfully forked to the Autoware Foundation repository.
- [ ] Some test data from people with consent are uploaded for testing.
- [ ] The SAM model's effectiveness in blurring faces and license plates is manually tested and confirmed.
- [ ] A decision is made on whether the anonymization tool will be standalone or integrated into autodistill.
- [ ] The tool is developed and tested to ensure it accurately anonymizes faces and license plates in rosbag camera data.
- [ ] Documentation is created to guide users on how to use the tool.
In last situation we decide won't use autodistill anymore because it don't have any additional things from original DINO and SAM for us.
Instead of this, we used Grounding DINO and SAM from original repositories and we added a image classification method OpenCLIP to validate Grounding DINO results. The working scheme is as follows:
Project Link: https://github.com/leo-drive/rosbag2_anonymizer
- The tool reads the images from a ROS2 bag file and feeds them to the Grounding DINO with original images. Grounding DINO takes an image and a list of prompts as input and finds the objects described by the prompts in the image.
- Grounding DINO cannot find all objects properly, so we validate objects with the image classification model OpenCLIP. OpenCLIP takes objects and a list of prompts as input and returns a matching score for the object and prompts.
- After validating the bounding box results, SAM segments all objects and finds masks. Then, the segmentation results are blurred and written to a new bag file.
Additional of these things, we want to add one more validation part. The new validation method will check whether some objects should be inside other objects or not. For example, a license plate should be inside of the car but should not be inside of human.
Also I will add detailed outputs, for now you can check this bag file which anonymized with our tool:
- https://drive.google.com/drive/folders/1jUZw7Um3b-I_ivYrOWdBSnHTqWnZd7cL?usp=sharing
Definition of done
- [ ] Add new validation part which checks the object positions
- [ ] Add detailed outputs
@StepTurtle could you test https://github.com/knzo25/rosbag2_language_sam with the same data and compare them?
cc. @knzo25
I'm expecting the comparison to be a playback of the anonymized rosbag camera image which as a video shared here.
@StepTurtle could you test https://github.com/knzo25/rosbag2_language_sam with the same data and compare them?
cc. @knzo25
I'm expecting the comparison to be a playback of the anonymized rosbag camera image which as a video shared here.
Here is the results:
- https://youtu.be/7YeGVJ9ulL8
- https://youtu.be/dQJeWQoZzec (one of the tool has low hz, maybe you can ignore this video)
In the video:
Left rqt window shows this tool: https://github.com/leo-drive/rosbag2_anonymizer
- 🔴 red boxes represents
license plate - 🟢 green boxes represents
human faces
Right rqt window shows this tool: https://github.com/knzo25/rosbag2_language_sam
- 🟣 purple boxes represents
license plate - 🟢 green boxes represents
cars
Additionally, a validation component has been added to https://github.com/leo-drive/rosbag2_anonymizer to verify the object positions. You can view the results here:
- https://youtu.be/xs3m2SlBnBg
Do you have any ideas or suggestions on what we can do in the upcoming stages?
I can read the text, blur is not enough.
There are so many places where the plates are not blurred well enough.
What happens if you look for license plates with low score threshold and if the plate is inside the vehicle for validation?
Blurring classes
Classes for the license plate detection
Parent classes
car
bus
truck
minibus
motorcycle
trailer
utility vehicle
tractor
golf cart
semi-truck
moped
scooter
Child class
license plate
Classes for the pedestrian face detection
Parent classes
person
child
Child class
human face
@xmfcx
I can read the text, blur is not enough.
There are so many places where the plates are not blurred well enough.
I changed the blur parameters, I guess it is okey now.
- https://youtu.be/w6lR1x4jtVg (with your child - parent labels)
For this question following schema could be helpful
The first step of validation involves running OpenClip. OpenClip will return results similar to the following:
- Assuming you have input prompts such as: ["license plate", "car", "face"]
- The output will look like this: [0.95, 0.4, 0.1]
If the score for the corresponding label is greater than 0.9, it will be selected as valid.
In the second validation step, we verify whether the label is inside of the parent. If it resides within one of the parent categories, it must satisfy one of the following conditions:
- Is the score for the corresponding label the highest among the scores?
- Is the score greater than 0.3?
What happens if you look for license plates with low score threshold and if the plate is inside the vehicle for validation?
For your example, license plate must have score greater than 0.3 or the highest score for the corresponding label the highest among the scores.
photo attribution from unsplash
My problem is with the false negatives, also known as, missed detections.
Does your proposal reduce FNs?
When we implemented this proposal, it didn't have a direct impact on FNs, but it allowed us to lower DINO threshold.
By reducing DINO threshold, we're able to detect more objects, including some that were previously classified as FNs. Also reducing DINO threshold will return a lot of FP and we aim to determine these FPs with proposal
@StepTurtle We can put the repository under AWF GitHub organization. Please make sure that you are not violating the license term of all the codes/models that you used.
@StepTurtle We can put the repository under AWF GitHub organization. Please make sure that you are not violating the license term of all the codes/models that you used.
@mitsudome-r @xmfcx we forked repository couple time ago.
But currently, I don't have write access. Could you give me a access to this repository? I believe I can create PRs, but I would prefer to push directly to the main branch as there might not be anyone to review for now. If this isn't acceptable, I'll create a PR whenever I need to update the code.
I am sharing the videos which shows the current results:
- https://www.youtube.com/watch?v=jfgAb6h4mmk
- https://www.youtube.com/watch?v=wQCBmNyj2gI
After labeling and training YOLOv8, we combined YOLOv8 and DINO to find bounding boxes and results improved.
Hi @xmfcx,
The tool have usage instructions in the project README. Should we also add a user guideline for the tools in the Autoware documentation. And instruction for how to publish new public dataset with Autoware community.
@StepTurtle under here: https://autowarefoundation.github.io/autoware-documentation/main/datasets/
it would be nice to have a separate page, dedicated to data anonymization.
Hi @StepTurtle, first of all, thank you for the tool you prepared. I had the opportunity to test the tool and I would like to give feedback about it. The tool version of I used:
- https://github.com/autowarefoundation/autoware_rosbag2_anonymizer
I used the tool to anonymize the data I collected in our autonomous test vehicle. You can find detailed information about the vehicle and system here:
- https://github.com/autowarefoundation/autoware.universe/issues/8114
ECU of Test Vehicle
The ECU specs of our test vehicle are as follows:
| Complement | Product |
|---|---|
| CPU | AMD Ryzen Threadripper PRO 3975WX 32-core, 64-thread |
| Memory | 256 GB RAM |
| GPU | 3x NVIDIA RTX A4000 (operations are performed on a single GPU) |
Anonymizing The Data
I anonymized the image in the bag file in the system that has the features I mentioned above. You can find the information about the bag file below:
| Property | Value |
|---|---|
| Bag size | 3.3 GiB |
| Storage id | sqlite3 |
| Duration | 116.724427755s |
| Total Messages | 939565 |
| Total Number of Topics | 314 |
| Image Message Rate | ~10 Hz |
| Image Message Type | sensor_msgs/msg/CompressedImage |
| Image Message Count | 1101 |
| Image Resolution (height x width) | 1860 x 2880 |
While anonymizing the data I provided above with the tool, the whole process took approximately 1 hour and 55 minutes. When I observed the approximate GPU usage with the nvidia-smi command throughout the process, I got the following result:
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 4498 G /usr/lib/xorg/Xorg 300MiB |
| 0 N/A N/A 31173 G ...seed-version=20240904-180241.692000 186MiB |
| 0 N/A N/A 313763 C python3 6830MiB |
| 1 N/A N/A 4498 G /usr/lib/xorg/Xorg 4MiB |
| 2 N/A N/A 4498 G /usr/lib/xorg/Xorg 4MiB |
+---------------------------------------------------------------------------------------+
Fri Sep 6 15:17:27 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA RTX A4000 On | 00000000:2E:00.0 On | Off |
| 71% 89C P2 68W / 140W | 7416MiB / 16376MiB | 100% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA RTX A4000 On | 00000000:41:00.0 Off | Off |
| 47% 66C P8 17W / 140W | 13MiB / 16376MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA RTX A4000 On | 00000000:61:00.0 Off | Off |
| 49% 67C P8 20W / 140W | 13MiB / 16376MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
Results and Observations
You can find images of anonymized data in this video:
- https://youtu.be/zMn2gRl7-2g
In conclusion, as can be observed from the video, the anonymization results are good enough. But the anonymization process took about 1 hour and 55 minutes. Considering the 116s total bag duration, this process time is not short and during this time, GPU usage was quite high.
@StepTurtle @xmfcx I have approved and merged the PR to Autoware Documentation. https://github.com/autowarefoundation/autoware-documentation/pull/557
Should we close this issue? If we want to do additional task from Nilay's feedback, we could consider creating a follow up issue (something like "make rosbag anonymizer tool faster")