vtr-verilog-to-routing
vtr-verilog-to-routing copied to clipboard
verilog-to-routing
How to run the Koios benchmarks for QoR
Expected Behaviour
We should have a clearly documented procedure for running the Koios benchmarks for QoR.
Current Behaviour
It appears the benchmarks are split over vtr_reg_nightly6, vtr_reg_weekly and vtr_reg_nightly4. The documentation at https://docs.verilogtorouting.org/en/latest/README.developers/#example-koios-benchmarks-qor-measurement just says to run vtr_reg_nightly6 and vtr_reg_weekly.
Possible Solution
If those are the only benchmarks we care about, we should say so and say why. If we care about all of them, we should have vtr_reg_nightly4 in the list too.
Another minor thing to fix: the comment in the code snippet at https://docs.verilogtorouting.org/en/latest/README.developers/#example-koios-benchmarks-qor-measurement says "run the VTR benchmarks". It should say "Run the Koios benchmarks"
Context
Generating results for a paper and making sure we have the full Koios benchmark suite run in the proper way.
Your Environment
VTR master as of Sept. 25.
We do want all of them. Will fix the QoR documentation.
Looks like the documentation at https://github.com/verilog-to-routing/vtr-verilog-to-routing/tree/master/vtr_flow/benchmarks/verilog/koios also needs to be fixed.
@aman26kbm As I have made a mistake while running koios (I was running part of it since I didn't know about all tasks), can we at least mention in the documentations why did we split it? If you agree, I can modify all documents (including Vaughn's suggestion).
The reason for the split was that the older Koios benchmarks could run with the default ODIN-only synthesis flow, but most of the newer Koios 2.0 benchmarks needed the Yosys+ODIN flow. So, we created new regression suites/tests for the ones that needed the Yosys+ODIN flow (vtr_reg_nightly6 and in vtr_reg_weekly). They differed in the cmake arg "ODIN_USE_YOSYS" (in .github/workflows/test.yml).
https://github.com/verilog-to-routing/vtr-verilog-to-routing/blob/37502f5b6ad0c6882455135374693291c540d79b/.github/workflows/test.yml https://github.com/verilog-to-routing/vtr-verilog-to-routing/pull/2165#issuecomment-1264239715
I agree this should have been clarified properly in the documentation.
I see that things have evolved since then. There is a new default flow called parmys. I don't see the ODIN_USE_YOSYS thing in test.yml anymore. I see that we now have separate regressions with "_odin" in their name (with a new cmake arg added to those i.e. WITH_ODIN=ON). These are the ones that worked with ODIN only.
I don't see weekly tests in test.yml though. Please check the settings for those.
Thanks for offering to fix the documentation, @saaramahmoudi. Appreciate it.