OpenROAD icon indicating copy to clipboard operation
OpenROAD copied to clipboard

Published 20 hours ago • verified publisher icon The-OpenROAD-Project

`repair_antennas` crash

Open kareefardi opened this issue 1 year ago • 12 comments

Describe the bug

Crash during running repair_antennas.

Expected Behavior

The command passing or a meaningful exit.

Environment

Git commit: 861b9b3822bd9f3d329a5f21e1795a1cf8ce7ce5
kernel: Linux 6.2.0-39-generic
os: Ubuntu 22.04.3 LTS (Jammy Jellyfish)
cmake version 3.25.1
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- OpenROAD version: v2.0-11613-g861b9b382
-- System name: Linux
-- Compiler: GNU 11.4.0
-- Build type: RELEASE
-- Install prefix: /usr/local
-- C++ Standard: 17
-- C++ Standard Required: ON
-- C++ Extensions: OFF
-- The C compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Performing Test C_COMPILER_SUPPORTS__-Wall
-- Performing Test C_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format -CMake Warning at src/CMakeLists.txt:241 (message):
  spdlog: SPDLOG_FMT_EXTERNAL=ON


 Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test C_COMPILER_SUPPORTS__-x
-- Performing Test C_COMPILER_SUPPORTS__-x - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-x
-- Performing Test CXX_COMPILER_SUPPORTS__-x - Failed
-- Performing Test C_COMPILER_SUPPORTS__c++
-- Performing Test C_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__c++
-- Performing Test CXX_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- TCL readline library: /usr/lib/x86_64-linux-gnu/libtclreadline.so
-- TCL readline header: /usr/include/x86_64-linux-gnu
-- Found SWIG: /usr/bin/swig4.0 (found suitable version "4.0.2", minimum required is "3.0")  
-- Found Boost: /usr/local/lib/cmake/Boost-1.81.0/BoostConfig.cmake (found version "1.81.0")  
-- boost: 1.81.0
-- Found Python3: /usr/include/python3.10 (found version "3.10.12") found components: Development Development.Module Development.Embed 
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 
-- spdlog: 1.9.2
-- Found BISON: /usr/bin/bison (found version "3.8.2") 
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- STA version: 2.4.0
-- STA git sha: 5b374dd36ad345c9fcd5224e9fc20484393568ab
-- System name: Linux
-- Compiler: GNU 11.4.0
-- Build type: RELEASE
-- Build CXX_FLAGS: -O3 -DNDEBUG
-- Install prefix: /usr/local
-- Found FLEX: /usr/bin/flex (found version "2.6.4") 
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- SSTA: 0
-- STA executable: /home/karim/work/OpenROAD/src/sta/app/sta
-- Found re2: /home/karim/Downloads/or-tools_x86_64_Ubuntu-22.10_cpp_v9.5.2237/lib/cmake/re2/re2Config.cmake (found version "9.0.0") 
-- Found Clp: /home/karim/Downloads/or-tools_x86_64_Ubuntu-22.10_cpp_v9.5.2237/lib/cmake/Clp/ClpConfig.cmake (found version "1.17.7") 
-- Found Cbc: /home/karim/Downloads/or-tools_x86_64_Ubuntu-22.10_cpp_v9.5.2237/lib/cmake/Cbc/CbcConfig.cmake (found version "2.10.7") 
-- Found SCIP: /home/karim/Downloads/or-tools_x86_64_Ubuntu-22.10_cpp_v9.5.2237/lib/cmake/scip/scip-config.cmake (found version "8.0.1") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- GPU is not enabled
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- Found Eigen3: /usr/share/eigen3/cmake/Eigen3Config.cmake (found version "3.4.0") 
-- GUI is enabled
-- Charts widget is not enabled
-- Found Boost: /usr/local/lib/cmake/Boost-1.81.0/BoostConfig.cmake (found version "1.81.0") found components: serialization 
-- Could NOT find VTune (missing: VTune_LIBRARIES VTune_INCLUDE_DIRS) 
-- Found Boost: /usr/local/lib/cmake/Boost-1.81.0/BoostConfig.cmake (found suitable version "1.81.0", minimum required is "1.78")  
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- Found Boost: /usr/local/lib/cmake/Boost-1.81.0/BoostConfig.cmake (found version "1.81.0") found components: serialization system thread 
-- TCL readline enabled
-- Tcl Extended disabled
-- Python3 enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/tmp.Et8pGUvhM6

To Reproduce

reproducible_antenna_bug.aa.tar.gz reproducible_antenna_bug.ab.tar.gz reproducible_antenna_bug.ac.tar.gz reproducible_antenna_bug.ad.tar.gz

cat reproducible_antenna_bug* > issue.tar.gz
tar xvf issue.tar.gz
cd reproducible_antenna_bug
bash run.sh

This is created from OpenLane2 since there is no template for OpenLane2 at the moment. OpenROAD .tcl scripts are very similar to one's used in OpenLane.

Relevant log output

[INFO GRT-0018] Total wirelength: 3425387 um
[INFO GRT-0014] Routed nets: 38969
[INFO GRT-0006] Repairing antennas, iteration 1.
[INFO GRT-0043] No OR_DEFAULT vias defined.
[WARNING ANT-0009] Net __dut__.__uuf__.n12568 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net696 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net711 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net725 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net815 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net824 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net826 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net829 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net834 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net852 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net1464 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net1999 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net2005 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net2007 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net2009 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net2036 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net2053 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net2055 requires more than 10 diodes per gate to repair violations.
[WARNING ANT-0009] Net net2213 requires more than 10 diodes per gate to repair violations.
[INFO GRT-0012] Found 286 antenna violations.
[INFO GRT-0015] Inserted 2078 diodes.
[INFO GRT-0054] Using detailed placer to place 82 diodes.
[INFO GRT-0009] rerouting 30474 nets.
[INFO GRT-0001] Minimum degree: 2
[INFO GRT-0002] Maximum degree: 51
[INFO GRT-0006] Repairing antennas, iteration 2.
[INFO GRT-0043] No OR_DEFAULT vias defined.
[INFO GRT-0012] Found 64 antenna violations.
[INFO GRT-0015] Inserted 353 diodes.
[INFO GRT-0054] Using detailed placer to place 1 diodes.
[INFO GRT-0009] rerouting 942 nets.
[INFO GRT-0001] Minimum degree: 2
[INFO GRT-0002] Maximum degree: 43
[INFO GRT-0006] Repairing antennas, iteration 3.
[INFO GRT-0043] No OR_DEFAULT vias defined.
Signal 11 received
Stack trace:
 0# 0x00005638C5AA2BFF in openroad
 1# 0x00007FBBBEC55A70 in /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libc.so.6
 2# odb::dbBox::getBox() in openroad
 3# odb::dbWirePathItr::getNextShape(odb::dbWirePathShape&) in openroad
 4# odb::tmg_conn::loadWire(odb::dbWire*) in openroad
 5# odb::tmg_conn::analyzeNet(odb::dbNet*) in openroad
 6# odb::orderWires(utl::Logger*, odb::dbNet*) in openroad
 7# grt::RepairAntennas::makeNetWire(odb::dbNet*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >&, std::map<int, odb::dbTechVia*, std::less<int>, std::allocator<std::pair<int const, odb::dbTechVia*> > >&) in openroad
 8# grt::RepairAntennas::makeNetWires(std::map<odb::dbNet*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >, grt::cmpById, std::allocator<std::pair<odb::dbNet* const, std::vector<grt::GSegment, std::allocator<grt::GSegment> > > > >&, int) in openroad
 9# grt::RepairAntennas::checkAntennaViolations(std::map<odb::dbNet*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >, grt::cmpById, std::allocator<std::pair<odb::dbNet* const, std::vector<grt::GSegment, std::allocator<grt::GSegment> > > > >&, int, odb::dbMTerm*, float) in openroad
10# grt::GlobalRouter::repairAntennas(odb::dbMTerm*, int, float) in openroad
11# 0x00005638C608F1D0 in openroad
12# TclNRRunCallbacks in /nix/store/2wmwl9g0204202qpj7p53w56rmi76m92-tcl-8.6.13/lib/libtcl.so
13# 0x00007FBBC2CCD228 in /nix/store/2wmwl9g0204202qpj7p53w56rmi76m92-tcl-8.6.13/lib/libtcl.so
14# Tcl_EvalEx in /nix/store/2wmwl9g0204202qpj7p53w56rmi76m92-tcl-8.6.13/lib/libtcl.so
15# Tcl_Eval in /nix/store/2wmwl9g0204202qpj7p53w56rmi76m92-tcl-8.6.13/lib/libtcl.so
16# sta::sourceTclFile(char const*, bool, bool, Tcl_Interp*) in openroad
17# ord::tclAppInit(Tcl_Interp*) in openroad
18# Tcl_MainEx in /nix/store/2wmwl9g0204202qpj7p53w56rmi76m92-tcl-8.6.13/lib/libtcl.so
19# main in openroad
20# 0x00007FBBBEC3FFCE in /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libc.so.6
21# __libc_start_main in /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libc.so.6
22# _start in openroad
run.sh: line 97: 2647081 Segmentation fault      (core dumped) openroad -exit -no_splash -metrics ./or_metrics_out.json ./scripts/openroad/antenna_repair.tcl

Screenshots

No response

Additional Context

No response

kareefardi avatar Jan 02 '24 15:01 kareefardi

@kareefardi I've tried it with the latest master branch, and the error is not happening anymore. Today we have merged a fix in GRT for pins not being properly covered, and my guess is that this crash was happening because of it.

Could you try with the latest master branch and see if it is fixed for you too?

eder-matheus avatar Jan 02 '24 21:01 eder-matheus

@eder-matheus I am still facing a crash using commit 0497bd0dac40e847ae0c846d3f1b7a098b95d918. out.log I attached the full log output in case its helpful.

kareefardi avatar Jan 03 '24 15:01 kareefardi

@kareefardi I've tried it again, running with gdb, and I still can't reproduce the crash. I'm trying now with valgrind.

eder-matheus avatar Jan 04 '24 15:01 eder-matheus

@eder-matheus there seems to be difference in reproducibility related to underlying architecture. I will let you know what I find.

kareefardi avatar Jan 09 '24 17:01 kareefardi

@kareefardi any progress or should we close this?

maliberty avatar Jan 29 '24 04:01 maliberty

@maliberty Sorry for the late update. We were able to reproduce this on two amd64 machines and not reproduce it on aarch64 machine. I also tried a debug build on the amd64 machine but I still got a crash. I am not sure yet if this is OS specific or not. @eder-matheus what kind of machine were you using when you test the crash?

kareefardi avatar Jan 29 '24 14:01 kareefardi

However, let me test the latest version and check if it is still the case.

kareefardi avatar Jan 29 '24 14:01 kareefardi

@eder-matheus what kind of machine were you using when you test the crash?

@kareefardi I've tested it in a MacOS with Intel and a CentOS with Intel. Unfortunately, I don't have access to an amd64 machine.

eder-matheus avatar Jan 29 '24 22:01 eder-matheus

Try running it under ASAN and UBSAN

QuantamHD avatar Jan 29 '24 23:01 QuantamHD

or plain old valgrind

maliberty avatar Jan 29 '24 23:01 maliberty

@kareefardi Could you try this branch: https://github.com/The-OpenROAD-Project/OpenROAD/pull/4635? I recently fixed a crash in one of our designs. Perhaps this fix works for your case as well.

eder-matheus avatar Feb 07 '24 14:02 eder-matheus

@eder-matheus Ok I will give it a try.

kareefardi avatar Feb 12 '24 16:02 kareefardi

@eder-matheus sorry for the delay. I can no longer reproduce the crash using the recent updates.

kareefardi avatar Feb 29 '24 19:02 kareefardi