commaai
build: speedup
@deanlee interesting in checking out why some objects take so long to build?
time g++ -v -ftime-report -o opendbc/can/dbc.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/threepilot/opendbc_repo/opendbc/dbc"' -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/home/batman/.pyenv/versions/3.11.4/include/python3.11 opendbc/can/dbc.cc
real 0m10.028s
user 0m7.034s
sys 0m2.953s
Time variable usr sys wall GGC
phase parsing : 0.91 ( 30%) 0.95 ( 52%) 1.86 ( 38%) 158925 kB ( 38%)
phase lang. deferred : 0.47 ( 15%) 0.29 ( 16%) 0.76 ( 16%) 82164 kB ( 20%)
phase opt and generate : 1.61 ( 52%) 0.57 ( 31%) 2.18 ( 44%) 171455 kB ( 41%)
|overload resolution : 0.41 ( 13%) 0.25 ( 14%) 0.59 ( 12%) 57574 kB ( 14%)
template instantiation : 0.71 ( 23%) 0.50 ( 28%) 1.25 ( 26%) 129017 kB ( 31%)
Due to the inefficiency of std::regex, it often takes over 90% of compile time (e.g., 10 seconds on my laptop). Many discussions online highlight that std::regex is slow both to compile and at runtime. Aside from switching to a more efficient regex library, there are few good solutions.
Also, this only takes 3s with all the regex:
batman@workstation-shane:~/openpilot/opendbc_repo$ time g++ dbc_test.cc -g -O2
real 0m3.445s
user 0m3.187s
sys 0m0.246s
It appears just adding an empty regex in the real dbc.cc causes a 2x slowdown, and subsequent patterns take almost no extra compile time.
Why does removing -g speed it up significantly?
std::regex generates a large amount of complex template code, leading to template bloat, which causes the compiler to take more time to process. This is especially true when the -g flag is enabled to generate debug information, as the compiler needs to produce detailed debug data for all the expanded templates.
RE2 looks like a good alternative to std::regex, it's has minimal headers and relatively small lib compare to other reg lib. https://github.com/google/re2
Can we precompile the regex stuff and check it in? I think another library is not worth it (unless super small), but it might be worth having to check in a new pre-compiled header when it changes.
Clang compiles template-heavy code like std::regex more efficiently than g++, potentially improving DBC compile times by up to 30%. However, the core issue of std::regex's complex template instantiations remains. Precompiling dbc.cc into platform-specific libraries (e.g., /precompiled/Darwin, /precompiled/arm64) may help but adds complexity to the build process and feels somewhat unusual. A more straightforward solution could be replacing std::regex with a faster alternative like Google's RE2, which is optimized for both performance and faster compilation.
#1602 is a PR to replace standard regex with RE2 library. When combined with compiling without optimization (-O0) the speed-up is significant (~5x speed improvement)
