vivado-risc-v
vivado-risc-v copied to clipboard
eugene-tarassov
Effort involved to port to Vitis flow
Hi @eugene-tarassov ,
Thank you for pushing the regular updates and expanding to newer boards.
I want to emulate a quad core large boom configuration on U250. I setup the toolchain and was able to generate the bitstream (timing met) on my laptop. I have access to U250 card at my school, however, due to some reasons, the environment only supports Vitis flow and thus I can not flash the card with the vivado flow generated mcs file.
May I check with you the effort involved to port it to Vitis? Following are steps I would follow:
- Once the Verilog generation from chisel is completed, create RTL kernel (.xo file) for whole verilog exposing two ports for vitis linking step. (i) First port is DDR connection. (ii) second port is is ethernet-U250.
- DDR connection is straight-forward; I can use the config file and link DDR to the RTL kernel.
- ethernet-U250 is the tricky part I feel. I'm thinking to use TCP-IP as the template and port to Vitis.
Do you see any potential/ future challenges I may face once I start to port to Vitis. If there are other better approaches or any thoughts you have, please suggest.
Thanks, Yash
PS: I have some experience working on RTL kernels, and implementing the host software for Vitis flow.
I have no experience with RTL kernels. It should be doable, but it looks like a lot of work.
Can you please clarify on the networking cables and switch between a desktop PC and VCU1525 card for NFS boot. #257 , you mentioned to use 10G SFP+ router and 40G QSFP+ to 4 x 10G SFP+ Breakout Cable for U250. Would the same setup work for VCU1525 card as well? I'm considering 10G SPF+ and 40G QSFP+ Breakout Cable. Are these fine?
Thanks. Yash
Thank you for the suggestion. I will order 10Gtek cable.
What are the changes in CONFIG_BOOTARGS= if I have to use static IP instead of ip=dhcp in Makefile refereed at https://github.com/eugene-tarassov/vivado-risc-v/blob/39a635ac5db0f43a33790345ac9a76a0546b5153/Makefile#L132C1-L135C207
Using the following format for static IP assignment: ip=<static_ip>::<gateway>:<netmask>:<hostname>:<interface>:<autoconf> , I updated ip to consider static IP and subnet mask /24, ip=192.168.0.243::192.168.0.1:255.255.255.0:<hostname>:eth0:none. What is the value for hostname? I'm not sure if it is debian or vcu1525-riscv ? Are the rest of the values for ip fine?
Thanks.
I'm not sure if it is debian or vcu1525-riscv?
debian
Are the rest of the values for ip fine?
It looks OK to me.
I was able to successfully generate the bitstream, and corresponding mcs file and prm file for rocket64b1. However I can not see "Hart #0" in xsdb. Following are the steps I followed:
- perform 'flash' with `make -f Makefile CONFIG=rocket64b1 BOARD=vcu1525 ROOTFS_URL=192.168.0.100:/home/nfsroot ROOTFS=NFS JTAG_BOOT=1 flash. Following is the log and flash seems to be successful.
xsdb -quiet board/jtag-freq.tcl
rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems.
attempting to launch hw_server
****** Xilinx hw_server v2020.1
**** Build date : May 27 2020 at 20:33:44
** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.
INFO: hw_server application started
INFO: Use Ctrl-C to exit hw_server application
INFO: To connect to this hw_server instance use url: TCP:127.0.0.1:3121
env HW_SERVER_ADDR=127.0.0.1:3121 \
env CFG_PART=mt25qu01g-spi-x1_x2_x4 \
env mcs_file=bitstream/vcu1525-riscv.mcs \
env prm_file=bitstream/rocket64b1/vcu1525-riscv.prm \
env XILINX_LOCAL_USER_DATA=no vivado -mode batch -nojournal -nolog -notrace -quiet -source board/program-flash.tcl
INFO: [Common 17-1239] XILINX_LOCAL_USER_DATA is set to 'no'.
INFO: [Labtools 27-2285] Connecting to hw_server url TCP:127.0.0.1:3121
INFO: [Labtools 27-2222] Launching hw_server...
INFO: [Labtools 27-2221] Launch Output:
****** Xilinx hw_server v2020.1
**** Build date : May 27 2020 at 20:33:44
** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.
INFO: [Labtools 27-3415] Connecting to cs_server url TCP:localhost:3042
INFO: [Labtools 27-3417] Launching cs_server...
INFO: [Labtools 27-2221] Launch Output:
****** Xilinx cs_server v2020.1.0
**** Build date : May 14 2020-09:10:29
** Copyright 2017-2020 Xilinx, Inc. All Rights Reserved.
INFO: [Labtoolstcl 44-466] Opening hw_target 127.0.0.1:3121/xilinx_tcf/Xilinx/12809621t269A
WARNING: [Xicom 50-38] xicom: No CseXsdb register file specified for CseXsdb slave type: 0, cse driver version: 0. Slave initialization skipped.
INFO: [Labtools 27-1434] Device xcvu9p (JTAG device index = 0) is programmed with a design that has no supported debug core(s) in it.
INFO: [Labtools 27-3164] End of startup status: HIGH
program_hw_devices: Time (s): cpu = 00:00:11 ; elapsed = 00:00:13 . Memory (MB): peak = 2687.719 ; gain = 0.000 ; free physical = 141 ; free virtual = 4351
INFO: [Labtools 27-2302] Device xcvu9p (JTAG device index = 0) is programmed with a design that has 1 SPI core(s).
Mfg ID : 20 Memory Type : bb Memory Capacity : 21 Device ID 1 : 0 Device ID 2 : 0
Performing Erase Operation...
Erase Operation successful.
Performing Blank Check Operation...
Blank Check Operation successful. The part is blank.
Performing Program and Verify Operations...
Program/Verify Operation successful.
INFO: [Labtoolstcl 44-377] Flash programming completed successfully
program_hw_cfgmem: Time (s): cpu = 00:00:05 ; elapsed = 00:16:51 . Memory (MB): peak = 2939.719 ; gain = 252.000 ; free physical = 248 ; free virtual = 4238
- power cycle the machine. Then, manually launch xsdb and run the following commands:
xsdb
rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems.
****** Xilinx System Debugger (XSDB) v2020.1
**** Build date : May 27 2020-20:33:44
** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.
xsdb% connect
tcfchan#0
xsdb% ta
1 xcvu9p
2 RISC-V JTAG DTM (Debug Transport Module timeout)
3 Legacy Debug Hub
xsdb% fpga bitstream/rocket64b1.bit
100% 30MB 1.6MB/s 00:18
xsdb% ta
1 xcvu9p
2 RISC-V JTAG DTM (Debug Transport Module timeout)
3 Legacy Debug Hub
xsdb% ta
1 xcvu9p
2 RISC-V JTAG DTM (Debug Transport Module timeout)
3 Legacy Debug Hub
xsdb% targets -set -filter {name =~ "Hart #0*"}
**no targets found with "name =~ "Hart #0*"". available targets:**
1 xcvu9p
2 RISC-V JTAG DTM (Debug Transport Module timeout)
3 Legacy Debug Hub
- I also tried with the bitstream (riscv-2020.2-vcu1525-64b2m.bit.gz) that was previously shared in #55 . Still, I do not see Hart 0. Following is the log:
xsdb
rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems.
****** Xilinx System Debugger (XSDB) v2020.1
**** Build date : May 27 2020-20:33:44
** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.
xsdb% connect
tcfchan#0
xsdb% targets -set -filter {name =~ "xc*"}
xsdb% fpga bitstream/
riscv-2020.2-vcu1525-64b2m.bit rocket64y4.bit vcu1525-riscv.prm
rocket64b1.bit vcu1525-riscv.mcs
xsdb% fpga bitstream/riscv-2020.2-vcu1525-64b2m.bit
100% 32MB 1.7MB/s 00:19
xsdb% ta
1* xcvu9p
2 RISC-V JTAG DTM (Debug Transport Module timeout)
3 Legacy Debug Hub
xsdb%
- I also tried to using "Program Device" and later launched xsdb to check for Hart. Still, same issue.
- One difference I notice is that, I donot see DDR under below SysMon (System Monitor) like you see in https://github.com/eugene-tarassov/vivado-risc-v/issues/55#issuecomment-1157947885
Any pointers are helpful to debug the issue.
Thanks.
It looks like the board fails to initialize DDR memory controller. Possible reasons: missing, damaged or incompatible DIMM memory module. Note that if you have replaced original DIMM on the board with a different DIMM model, you need to modify DDR MC settings in the Vivado project.
I checked the DIMM model -- they are the original ones provided along the board. The part number I see is "MTA18ASF2G72PZ-2G3B1RG", and matches with the one in user guide "MTA18ASF2G72PZ-2G3B1" expect RG. I will try to setup XRT and deployment shell this week and validate the pcie link/ DMA/DDR tests to make sure the board is fine. Can you please share couple of bitstreams that you tested recently?.
Thanks.
Try to reboot the host machine after loading bitstream to FPGA: sudo reboot. I tried latest bitstream and it looks like some recent changes in the Linux kernel prevents it to communicate to the board after reloading bitstream and the board is left in reset state. I see same "Debug Transport Module timeout" error, but it works OK after reboot.
I setup the XRT and validated the card. Log seems to be fine.
yash@yash-SYS-5038K-I-ES1:~$ sudo /opt/xilinx/xrt/bin/xbutil validate
INFO: Found 1 cards
INFO: Validating card[0]: xilinx_vcu1525_xdma_201830_1
INFO: Checking PCIE link status: PASSED
INFO: Starting verify kernel test:
INFO: verify kernel test PASSED
INFO: Starting DMA test
Host -> PCIe -> FPGA write bandwidth = 9100.4 MB/s
Host <- PCIe <- FPGA read bandwidth = 12116.6 MB/s
INFO: DMA test PASSED
INFO: Starting DDR bandwidth test: ..........
Maximum throughput: 47596.750000 MB/s
INFO: DDR bandwidth test PASSED
INFO: Card[0] validated successfully.
INFO: All cards validated successfully.
Are you using commit: c1bc480a2e8340fa45b3dd3a39fd9c44ba279b60 and Vivado 2024.2 to generate the bitstream?
While debugging the vcu1525, I was checking sudo lspci -vvvand following is the log:
I see the Link speed 5GT/s and Width as x8 below in fields LnkSta, LnkCap. 5 GT/s seems to be PCIe Gen 2. Could this be the issue for the card to be reset state?
01:00.0 Memory controller: Xilinx Corporation Device 9028
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 11
NUMA node: 0
Region 0: Memory at fb800000 (64-bit, non-prefetchable) [size=256K]
Region 2: Memory at fb400000 (64-bit, non-prefetchable) [size=4M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [200 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
This is lspci -vvv output for a life, fully functional VCU1525 RISC-V design:
01:00.0 Memory controller: Xilinx Corporation Device 9028
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 11
Region 0: Memory at df400000 (64-bit, non-prefetchable) [disabled] [size=256K]
Region 2: Memory at df000000 (64-bit, non-prefetchable) [disabled] [size=4M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, NROPrPrP-, LTR-
10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-, TPHComp-, ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [200 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Thank you for sharing the lspci output. I notice that SlotPowerLimit value as 0 in my output vs SlotPowerLimit value as 75. I will check if I can change BIOS settings to update the slot power limit.
I changed the vivado version to launch hardware manager and xsdb from 2020.1 to 2022.2. Debug Transport Module timeout is now changed to Debug Transport Module is held in reset . Something is causing the card to get stuck in reset. I'm currently using a Supermicro server to power the card and a laptop for xsdb / uploading the bitstream. I will try with a different workstation and check if motherboard cause any issue.
xsdb
rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems.
****** Xilinx System Debugger (XSDB) v2022.2
**** Build date : Oct 14 2022-05:10:29
** Copyright 1986-2022 Xilinx, Inc. All Rights Reserved.
xsdb% connect
tcfchan#0
xsdb% ta
1 xcvu9p
2 RISC-V JTAG DTM (Debug Transport Module is held in reset)
3 Legacy Debug Hub
xsdb%
Just to be sure, can you please clarify on how do you program the card? On vcu1525 I only see three ports: two are QSFP and one is micro-usb interface. Do you use a standard usb to micro-usb cable ? In #55 , where you were referring to USB-JTAG, do you mean a standard usb to micro-usb cable?
I'm ordering two DDR4 DIMMs with the exact same part number from the user guide and try again the bitstream (riscv-2020.2-vcu1525-64b2m.bit.gz) you shared in #55. As I will be removing 4 existing DIMMs from the board, instead only use new ones. Do the new DIMMs go to two DIMM slots far from the PCIe interface i.e. slot farthest to card's PCIe interface is DIMM 0?
Meanwhile, I try to look into active reset problem you pointed in previous thread by mapping reset to board LEDs.
Also, I'm seeing following warning for VCU1525 when I upload the bitstream with HW manager.(No such warning for Genesys2). Did you see this warning uploading the bitstream?
WARNING: [Labtools 27-3361] The debug hub core was not detected.
Resolution:
1. Make sure the clock connected to the debug hub (dbg_hub) core is a free running clock and is active.
2. Make sure the BSCAN_SWITCH_USER_MASK device property in Vivado Hardware Manager reflects the user scan chain setting in the design and refresh the device. To determine the user scan chain setting in the design, open the implemented design and use 'get_property C_USER_SCAN_CHAIN [get_debug_cores dbg_hub]'.
For more details on setting the scan chain property, consult the Vivado Debug and Programming User Guide (UG908).
Do the new DIMMs go to two DIMM slots far from the PCIe interface
According to VCU1525 User Guide, C0 slot is close to PCIe:
I'm seeing following warning for VCU1525 when I upload the bitstream with HW manager
I use XSDB to load bitstream, it does not show any warnings.
While looking at pins used in the design, I notice that AU21 is floating. Could this be causing memory controllers / calibration not showing? Should this pin be driven to zero so that DDR is in normal operation?
I'm not sure if it is relevant, I see Rev 1.0 on the PCIe slot of vcu1525. Regarding the LEDs:
- DS1 (red): just flickers during the host machine boot and remain off.
- DS2 (blue): stays solid, always on.
- DS3 (Green): blinking always.
- DS3 (yellow): stay solid, always on. however, iif there is an existing mcs in flash memory. During cold boot, momentarily goes down (2-3 seconds) and back again.
- DS3 (red): Like DS1 red, flickers during the host machine boot and remain off.
To narrow down the issue, I was checking some internal signals this week by mapping them to the LEDs. Seems I can only control yellow LED of the three DS3 LEDS. So, I generated three different bitstreams mapping different internal signals to yellow LED. (1) pcie_resetn (2) resetn (3) mem_ok / init_calib_complete.
For pcie_resetn, LED stays solid -> correct. For resetn, LED stays solid -> correct. For init_calib_complete, LED off. -> issue.
Resets are working fine. Calibration is the issue. I will be collecting the new DIMMs this week. Apart from that, do you think anything else could be the issue?
Thanks.
@eugene-tarassov ,
I implemented an example with xdma-ddr4 IPs to isolate and understand DDR calibration issue. I see calibration pass as in the below image. The reason I think why I'm unable to successfully bringup RISC-V system might be due to a different card revision than the one you have.
Seems some clocks are not available on power-on for some revisions of the cards. please see https://adaptivesupport.amd.com/s/article/71680?language=en_US . I will try these suggestions and see if they can fix the issue.