AutoTuner fails to complete the flow

[vijayank88]

Subject

[Flow] for any util, flow Makefile, or flow script issues.

Describe the bug

@vvbandeira

I tried to run asap7/riscv32i by changing config.mk config.mk.txt and autotuner.json autotuner.json.txt

I kept empty "_FR_FILE_PATH": "" as ASAP7 platform don't have fastroute.tcl.

AutoTuner ends up with Best Parameters found.

[INFO TUN-0002] Best parameters found: {'_SDC_CLK_PERIOD': 1287.1798528615634, 'CORE_UTILIZATION': 72, 'CORE_ASPECT_RATIO': 1.1170376199877676, 'CORE_MARGIN': 2, 'CELL_PAD_IN_SITES_GLOBAL_PLACEMENT': 1, 'CELL_PAD_IN_SITES_DETAIL_PLACEMENT': 1, '_FR_LAYER_ADJUST': 0.27288320224415746, 'PLACE_DENSITY_LB_ADDON': 0.5910115630799327, '_PINS_DISTANCE': 1, 'CTS_CLUSTER_SIZE': 110, 'CTS_CLUSTER_DIAMETER': 396, '_FR_FILE_PATH': '', '_FR_GR_OVERFLOW': 1}
(save_best pid=20941) [INFO TUN-0003] Best parameters written to /home/vijayan/OPENROAD_FLOW/ORFS_LOCAL/OpenROAD-flow-scripts/flow/logs/asap7/riscv32i/test-tune-2023-08-29-07-36-20/autotuner-best-c35216c8.json

But if I check respective variant, failed at detail routing stage with below error.

[INFO DRT-0166] Complete pin access.
[INFO DRT-0267] cpu time = 00:01:51, elapsed time = 00:00:07, memory = 290.50 (MB), peak = 293.75 (MB)
[ERROR DRT-0155] Guide in net net114 uses layer M8 (16) that is outside the allowed routing range [M2 (4), M7 (14)] with via access on [M1 (2)].
Error: detail_route.tcl, 70 DRT-0155
Elapsed time: 0:11.25[h:]min:sec. CPU time: user 114.09 sys 0.35 (1016%). Peak memory: 300796KB.

In that case AutoTuner should not generate Best parameters found tag right?

Expected Behavior

If the flow completes till finish stage, then AutoTuner should tag Best parameters, otherwise print last run tag for reference to view and analyze for failures.

Environment

Git commit: ea76d302837cf6c9a2f2eca671880d441fe3400a
kernel: Linux 3.10.0-1160.90.1.el7.x86_64
os: CentOS Linux 7 (Core)
cmake version 3.24.2
-- The CXX compiler identification is GNU 8.3.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rh/devtoolset-8/root/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- OpenROAD version: v2.0-10032-gea76d3028
-- System name: Linux
-- Compiler: GNU 8.3.1
-- Build type: RELEASE
-- Install prefix: /usr/local
-- C++ Standard: 17
-- C++ Standard Required: ON
-- C++ Extensions: OFF
-- The C compiler identification is GNU 8.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/devtoolset-8/root/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /usr/local/bin/python3.9 (found version "3.9.6") found components: Interpreter 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Performing Test C_COMPILER_SUPPORTS__-Wall
-- Performing Test C_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test C_COMPILER_SUPPORTS__-x
-- Performing Test C_COMPILER_SUPPORTS__-x - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-x
-- Performing Test CXX_COMPILER_SUPPORTS__-x - Failed
-- Performing Test C_COMPILER_SUPPORTS__c++
-- Performing Test C_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__c++
-- Performing Test CXX_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- TCL library: /usr/lib64/libtcl.so
-- TCL header: /usr/include/tcl.h
-- TCL readline library: /usr/lib64/libtclreadline.so
-- TCL readline header: /usr/include
-- Found SWIG: /usr/bin/swig (found suitable version "4.1.0", minimum required is "3.0")  
-- Using SWIG >= 4.1.0 -flatstaticmethod flag for python
-- Found Boost: /usr/local/lib/cmake/Boost-1.72.0/BoostConfig.cmake (found version "1.72.0")  
-- boost: 1.72.0
-- Found Python3: /usr/local/include/python3.9 (found version "3.9.6") found components: Development Development.Module Development.Embed 
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7") 
-- spdlog: 1.8.1
-- Found BISON: /usr/bin/bison (found version "3.0.4") 
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") found components: doxygen dot 
-- STA version: 2.4.0
-- STA git sha: 7cf916ba205115a06c4531a044ced481f1ff8f12
-- System name: Linux
-- Compiler: GNU 8.3.1
-- Build type: RELEASE
-- Build CXX_FLAGS: -O3 -DNDEBUG
-- Install prefix: /usr/local
-- Found FLEX: /usr/local/bin/flex (found version "2.6.4") 
-- TCL library: /usr/lib64/libtcl.so
-- TCL header: /usr/include/tcl.h
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- SSTA: 0
-- STA executable: /home/vijayan/OPENROAD_FLOW/ORFS_LOCAL/OpenROAD-flow-scripts/tools/OpenROAD/src/sta/app/sta
-- GPU is not enabled
-- TCL library: /usr/lib64/libtcl.so
-- TCL header: /usr/include/tcl.h
-- Found re2: /opt/or-tools/lib64/cmake/re2/re2Config.cmake (found version "9.0.0") 
-- Found Clp: /opt/or-tools/lib64/cmake/Clp/ClpConfig.cmake (found version "1.17.7") 
-- Found Cbc: /opt/or-tools/lib64/cmake/Cbc/CbcConfig.cmake (found version "2.10.7") 
-- Found Eigen3: /usr/local/share/eigen3/cmake/Eigen3Config.cmake (found version "3.3.90") 
-- Found SCIP: /opt/or-tools/lib/cmake/scip/scip-config.cmake (found version "8.0.1") 
-- GUI is enabled
-- Charts widget is not enabled
-- Found Boost: /usr/local/lib/cmake/Boost-1.72.0/BoostConfig.cmake (found version "1.72.0") found components: serialization 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Could NOT find VTune (missing: VTune_LIBRARIES VTune_INCLUDE_DIRS) 
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found suitable version "1.80.0", minimum required is "1.78")  
-- TCL library: /usr/lib64/libtcl.so
-- TCL header: /usr/include/tcl.h
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0") found components: serialization system thread 
-- TCL readline enabled
-- Tcl Extended disabled
-- Python3 enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/tmp.HHWqOVNWE2

To Reproduce

Update config.mk and autotuner.json as per attachment in the description.

cd flow/util
python3.9 distributed.py --design riscv32i --platform asap7 --config ../designs/asap7/riscv32i/autotuner.json tune

Relevant log output

No response

Screenshots

No response

Additional Context

If I update "_FR_FILE_PATH": "../../../platforms/asap7/fastroute.tcl", by adding fastroute.tcl in respective path, AutoTuner completes the flow with all stages and able to get relevant results as per Best Parameter Found tagged by the flow.

[luarss]

Reproduced the failure with current python3 distributed.py --design riscv32i --platform asap7 --config ../designs/asap7/riscv32i/autotuner.json tune and provided files.

@vijayank88 If all the runs are a failure (returns some consistently large big number, in my case 1e+13) then the code is such that it will return the first trial where the failure occurred. If there has been a successful the lowest cost will be reported accordingly

@vvbandeira Do you think we should add a warning in the final message saying that this was a errored run?

[vvbandeira]

@luarss, yes. If all runs fail we should print an error message and distributed.py should exit with a non-zero value.