chipyard
chipyard copied to clipboard
ucb-bar
ASAP7 tutorial power-par error
Background Work
- [X] Yes, I searched the mailing list
- [X] Yes, I searched prior issues
- [X] Yes, I searched the documentation
Chipyard Version and Hash
Release: 1.6.2 Hash: 481398b
OS Setup
Linux edabox 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: RedHatEnterprise Description: Red Hat Enterprise Linux release 8.6 (Ootpa) Release: 8.6 Codename: Ootpa
Other Setup
Using genus 21.10, innovus 21.11 and voltus 21.12
Current Behavior
When running the power-par build target (make power-par CONFIG=TinyRocketConfig BINARY=$RISCV/riscv64-unknown-elf/share/riscv-tests/isa/rv32ui-p-simple) I get the following error
"sim.inputs.execution_flags": [ "+verbose", "+vcdplusfile=/home/odxa20/chipyard/vlsi/output/chipyard.TestHarness.TinyRocketConfig/rv32ui-p-simple.vpd", "+dramsim", "+dramsim_ini_dir=/home/odxa20/chipyard/generators/testchipip/src/main/resources/dramsim2_ini", "+max-cycles=10000000" ], "sim.inpmake: *** [/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/hammer.d:68: /home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/power-sim-par-input.json] Error 1
Expected Behavior
Should finish normally
Other Information
No response
At first glance, your error dump seems weirdly incomplete... can you post a few files first:
- The Makefile-generated
sim-inputs.ymlfile andsim-debug-inputs.yml - The Makefile-generated
hammer.d - The Hammer-generated
power-sim-par-input.jsonfile
Nothing appears to be wrong with the input files. Can you post the hammer-vlsi-<timestamp>.log file that corresponds to this error?
Of course. These are the last two log files that where output before the error (no newer log files were produced) hammer-vlsi-20220614-104059.log hammer-vlsi-20220614-104018.log
I just ran the command again (make power-par CONFIG=TinyRocketConfig BINARY=$RISCV/riscv64-unknown-elf/share/riscv-tests/isa/rv32ui-p-simple) after doing a make clean. The error message seems to be clearer
"sim.inputs.execution_flags": [
"+verbose",
"+vcdplusfile=/home/odxa20/chipyard/vlsi/output/chipyard.TestHarness.TinyRocketConfig/rv32ui-p-simple.vpd",
"+dramsim",
"+dramsim_ini_dir=/home/odxa20/chipyard/generators/testchipip/src/main/resources/dramsim2_ini",
"+max-cycles=10000000"
],
"sim.inpTraceback (most recent call last):
File "./example-vlsi", line 62, in
I am attaching all the .log files generated in this fresh run logs.zip
I see. It looks like Python is choking up trying to write the sim-to-par output to power-sim-par-input.json quickly followed by a print to stdout. It seems like some output buffer in your system is overfilled. We haven't seen this before on our systems, but this definitely needs fixing.
I think this is a case of old sloppy programming. Can you add f.close() between the f.write(output_str) and print(output_str) in your /home/odxa20/chipyard/vlsi/hammer/src/hammer-vlsi/hammer_vlsi/cli_driver.py at line 1283?
I made the change you suggested but got the same error
"synthesis.outputs.output_files": [
"/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/syn-rundir/ChipTop.mapped.v"
],
"synthesis.outputs.sdc": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/syn-rundir/ChipTop.mapped.sdc",
"synthesis.outputs.seq_cells": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/syn-rundir/find_regs_cells.json",
"synthesis.outputs.all_regs": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/syn-rundir/find_regs_paths.json",
"synthesis.outputs.sdf_file": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/syn-rundir/ChipTop.mapped.sdf",
"par.inputs.input_files": [
"/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/syn-rundir/ChipTop.mapped.v"
],
"par.inputs.top_module": "ChipTop",
"par.inputs.post_synth_sdc": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/syn-rundir/ChipTop.mapped.sdc",
"par.outputs.output_gds": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.gds",
"par.outputs.output_netlist": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.lvs.v",
"par.outputs.output_sim_netlist": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.sim.v",
"par.outputs.hcells_list": [],
"par.outputs.seq_cells": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/find_regs_cells.json",
"par.outputs.all_regs": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/find_regs_paths.json",
"par.outputs.sdf_file": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.par.sdf",
"par.outputs.spefs": [
"/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.PVT_0P63V_100C.par.spef",
"/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.PVT_0P77V_0C.par.spef"
],
"power.inputs.top_module": "ChipTop",
"power.inputs.netlist": "/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.lvs.v",Traceback (most recent call last):
File "./example-vlsi", line 62, in <module>
ExampleDriver().main()
File "/home/odxa20/chipyard/vlsi/hammer/src/hammer-vlsi/hammer_vlsi/cli_driver.py", line 1361, in main
sys.exit(self.run_main_parsed(vars(parser.parse_args(args))))
File "/home/odxa20/chipyard/vlsi/hammer/src/hammer-vlsi/hammer_vlsi/cli_driver.py", line 1284, in run_main_parsed
print(output_str)
BlockingIOError: [Errno 11] write could not complete without blocking
"power.inputs.spefs": [
"/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.PVT_0P63V_100C.par.spef",
"/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.PVT_0P77V_0C.par.spef"
]
}make: *** [/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/hammer.d:71: /home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/power-par-input.json] Error 1
I am attaching the log files
If this is due to a buffer being full is there a way to increase the buffer size (e.g with some kind of ulimit command ?) logs.zip
Looks like it got a little farther in the output print... but I see, the issue is actually not with the file write (power-sim-par-input.json in this case) but actually the print to stdout.
Based on this: https://stackoverflow.com/questions/65876753/how-to-make-pythons-print-non-blocking it looks like you can bypass output buffering with a flush=True in the print. ulimit seems overkill/dangerous just to get around this.
Regardless, I think the print(output_str) is quite unnecessary from a usability perspective because it pollutes the terminal buffer for these X-to-Y steps. Can you instead just replace that line with something like print("Output config written to " + args["output"]) and see if this error goes away?
This seems to have fixed it. Thank you very much. Unfortunately I am getting an other error.
Begin Processing Power Net/Grid for Power Calculation
Rail status:
0.63V VDD
0V VSS
Ended Processing Power Net/Grid for Power Calculation: (cpu=0:00:00, real=0:00:00, mem(process/total/peak)=2729.66MB/12962.07MB/7500.04MB)
Begin Processing Timing Window Data for Power Calculation
clock_clock(1000MHz)
Timing Window Statistics:
Waveform definitions parsed : 1
Clock roots assigned/parsed : 1/1 (100%)
Constants assigned/parsed : 1342/1342 (100%)
External loads assigned/parsed : 37/37 (100%)
Slews assigned/parsed : 123224/123224 (100%)
Nets with slew/Nets in design : 58532/58532 (100%)
Slew range : 1.375e-12s - 3.96375e-10s
SPEF : /home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/par-rundir/ChipTop.PVT_0P63V_100C.par.spef
Name matched: 58495/58495
Annotation coverage for this file : 58495/58532 (99.9368%)
Ended Processing Timing Window Data for Power Calculation: (cpu=0:00:01, real=0:00:00, mem(process/total/peak)=2750.31MB/12960.06MB/7500.04MB)
Begin Processing VCD Vectors
Parsing VCD file /home/odxa20/chipyard/vlsi/output/chipyard.TestHarness.TinyRocketConfig/rv32ui-p-simple.vpd
Starting Reading VCD variables
2022-Jun-15 21:37:11 (2022-Jun-15 18:37:11 GMT)
---------------------- FATAL ----------------------
Tcl command failed.
-->
** ERROR: (VOLTUS_POWR-1151): Voltus Power Analysis was unable to complete
during the processing of VCD.
The Error was:
** ERROR: (VOLTUS_POWR-1735): Unable to read the activity file because of the following error: {[/home/odxa20/chipyard/vlsi/output/chipyard.TestHarness.TinyRocketConfig/rv32ui-p-simple.vpd line 1, col 1] syntax error}. Fix the error and run power analysis again.
while executing
"ChipPwr fnExecute dynamic"
(file "/tmp/ssv_tmpdir_293729_8dWKPD/voltus_power_293729_3.cmd" line 74)
-------------------------------------------------------
Exit code 2.
Voltus Power Analysis exited unsuccessfully.
** ERROR: (VOLTUS_POWR-1390): Voltus Power Analysis could not complete successfully.
**ERROR: (IMPSE-110): File '/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/power-par-rundir/power.tcl' line 42: 1.
#@ End verbose source /home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/power-par-rundir/power.tcl
**ERROR: (UI-93): 1
These are the logs logs.zip
This is the vcd file in question rv32ui-p-simple.zip
Glad to see that fixed it - I'll make a PR to Hammer.
Hm, unfortunately the logs do not capture the run_simulation output log part of the VCS simulation, so I'm unable to see if there were any issues with the VPD writing. Did VCS report a successful waveform dump?
Also, you're using Voltus 21.12. Do you know if the read_activity_file command in this newer version distriguishes vpd as distinct from the vcd format? If so, that might explain the syntax error?
vcs did not produce an error when running. Is there a way to check the validity of the .vpd file ? We unfortunately don't currently have access to older versions of Voltus. Is there a way to check if the read_activity_file command in this newer version distinguishes vpd and vcd formats ?
You can try to read the VPD into a waveform viewer. You have a couple options here:
- If you still have licenses for DVE or WV, the VPD can be read directly
- If not, you can run the
vpd2vcdutility (same install dir asvcs) and then read it with Verdi or basically any other waveform viewer.
Yes, you can pull up either the Voltus Text Command Reference or in the Voltus terminal you can type help read_activity_file and it will list all the documentation.
Hi. I just checked the Voltus Text Command Reference and there is no mention of vpd files. The file types that are supported are VCD | TCF | SAIF | FSDB | PHY | SHM. I am attaching the documentation file below. I am in the process of getting DVE working since our installation did not have it installed by default so that I can check that the vpd trace file is valid. Again I am very grateful for your help and hope we can get this working read_activity_file.pdf
I just checked the Voltus Text Command Reference and there is no mention of vpd files.
In Voltus, VPD files are supposed to be read as VCD format. It would be odd if it no longer supports VPD...
In the attached file which I exported from the Voltus Text Command Reference there is no mention of vpd (maybe it is implied they are treated the same as vcd ?). Does your version explicilty mention vpd files ?
Right, for many versions now it is implied they are treated the same as VCD. I'm pretty sure it's built on VCD but is a compressed binary format.
You could also try dumping in FSDB format instead. However, I have noticed that Voltus' FSDB reader is a few versions behind VCS' FSDB writer, so you'd need to use an older version of VCS.
I just opened the vpd file with wv and it seems fine. No errors and signals are showing. What would be the next step to find the issue ?
I see. Can you try converting it to VCD and reading that into Voltus instead (override the waveforms key)
I just managed to get vpd2vcd working (was missing some 32-bit libraries) and converted the vpd file to vcd. What do you mean by override the waveforms key ? (sorry for my inexperience. this is the first time I am working with industrial cad tools)
The power.inputs.waveforms key (set via vlsi/Makefile into the power-inputs.yml file).
So I change line 189 of the vlsi/Makefile from echo "sim.outputs.waveforms: ['$(sim_out_name).vpd']" >> $@ to echo "sim.outputs.waveforms: ['~/rv32ui-p-simple.vcd']" >> $@ right ?
No, edit power.inputs.waveforms, otherwise you'll need to run the sim-to-power step again. This way you can directly run power-par.
Ok got it. Changed that and will run the power-par command. Once it's finished I will post the results
For some reason after running make power-par CONFIG=TinyRocketConfig BINARY=$RISCV/riscv64-unknown-elf/share/riscv-tests/isa/rv32ui-p-simple with the changed waveforms key in power-inputs.yml the entire process started from the start (synthesis). I am wondering whether this is due to the previous build ending in the voltus error and me having to CTRL-C out of it. Thus I resorted to changing the filename in the Makefile since the power.inputs.waveforms is overwritten when the build restarts
You can run the redo-power-par target. This removes all the previous flow dependencies.
I managed to run the power-par step with the converted vcd file. From my tests even when running redo-power-par the power-inputs.yml file gets overwritten so to do this I ran power-par once from scratch, then converted the file using vpd2vcd in the output folder and changed the extension in the Makefile to vcd and ran the redo-power-par command. Unfortunately more errors arise
Starting Calculating power
2022-Jun-16 22:35:21 (2022-Jun-16 19:35:21 GMT)
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT): 10%
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT): 20%
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT): 30%
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT): 40%
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT): 50%
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT): 60%
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT): 70%
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT): 80%
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT): 90%
Finished Calculating power
2022-Jun-16 22:35:22 (2022-Jun-16 19:35:22 GMT)
Ended Power Analysis: (cpu=0:00:05, real=0:00:00, mem(process/total/peak)=3666.79MB/13912.02MB/7728.89MB)
Begin Writing Current Files
This vectcorbased dynamic simulation is from 0 second to 0.01 second.
CPU : 24
Simulation Period: 1e+07ns
Step Size : 500ps
** WARN: (VOLTUS_POWR-1812): Provided step size is very small compared to the simulation period.
This may have a significant impact on runtime and output file size.
You may want to increase the step size for better performance.
Starting Processing toggles for instances
2022-Jun-16 22:35:28 (2022-Jun-16 19:35:28 GMT)
** ERROR: (VOLTUS_POWR-1390): Voltus Power Analysis could not complete successfully.
**ERROR: (IMPSE-110): File '/home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/power-par-rundir/power.tcl' line 42: 1.
#@ End verbose source /home/odxa20/chipyard/vlsi/build/chipyard.TestHarness.TinyRocketConfig-ChipTop/power-par-rundir/power.tcl
**ERROR: (UI-93): 1
I am attaching the power.tcl file produced since the error seems to occur at line 42 of said file
power.zip
Ok, so it looks like it was able to run the read_activity_file (line 40) with the VCD file instead of the VPD. I'll make an issue in hammer-cadence-plugins to investigate why the Cadence tools don't support VPD anymore.
From the log, it looks like it was able to analyze the power (it says it finished calculating) but I think the warning could reveal something. Your simulation seems excessively long (0.01 seconds, or 2e7 time steps) - did the simulation timeout, or do you have a custom program that you're running? I have a suspicion that Voltus is unable to process the toggles for so many instances over so many timestamps and may have run out of resources, hence the error.
The test program I am using is the one from the tutorial rv32ui-p-simple which is quite small. Nonetheless when looking at the resource monitor once the simulation starts the systems ram (16GB) is immediately filled and swap space starts to get used. After a few seconds I receive the error so your hypothesis about resources seems to be correct. How can this be fixed ?
We need to fix the simulation timeout first so that your waveforms aren't 0.01s long. Unfortunately, there is a fix for ASAP7 SRAM behavioral models (ucb-bar/hammer#645) that hasn't yet made it into the Chipyard release. Without these fixes, certain simulations fail and thus timeout.
Can you update all of your Hammer and plugins repositories to their latest commits on master branch, clean, and try your flow again?
Should I delete and clone the repositories from scratch or try to do a merge ?