CGAL
Fix undefined behavior sanitizer casting error in Arrangement_2_iterators.h
Please use the following template to help us managing pull requests.
Summary of Changes
Fixed runtime downcast errors detected by the Undefined Behavior Sanitizer (UBSan) in Arrangement_2_iterators.h.
The code was using a functional cast pointer(p), which acts like a static_cast. This caused undefined behavior when the object at p was not fully initialized as the derived type. I replaced these with reinterpret_cast
Release Management
Affected package(s): Arrangement_on_surface_2 Issue(s) solved (if any): fix #9140 Feature/Small Feature (if any): Link to compiled documentation: N/A License and copyright ownership: I agree to transfer the copyright of my contributions to the CGAL project owner.
The reinterpreting cast seems a bit harsh. It does not fix the issue, but hide it, in my opinion. Why do you think those pointers are valid to be casted that way?
@lrineau Thank you for the review. You are right that reinterpret_cast is aggressive.
My reasoning for using it was based on the UBSan error log. The log showed that the object at address p exists and is of type CGAL::Arr_face (the internal representation), but the iterator is trying to cast it to value_type (the external Face type).
The original functional cast pointer(p) acts like a static_cast, which requires the runtime object to technically be the derived type. Since the object seems to be only instantiated as the base Arr_face, UBSan flagged it.
I assumed that Arr_face and the external Face type are layout-compatible here, so treating the pointer as the derived type would be safe in practice.
If there is a safer, standard way in CGAL to convert these internal representation pointers to their external handles without triggering UB, I would be happy to implement that instead. Do you have a suggestion for a safer alternative?
Hi @lrineau , just a gentle follow-up on this.
I wanted to check if there are any changes required from my side or if the approach looks okay? I am happy to make further corrections if needed.
Thanks for your time!