Plans to support 2020.1?

[rsarwar87]

Hi - Not really an issue, but I just want to start a discussion. I was wondering if there are any plans to move the linux-kernels/uboot repos to 2020.1? the fpga manage has matured so loading bitstreams and clocks using overlays is now possible.

I have been porting the sdk to use 2020.1 and it is working well with ultra-scale devices (not tested zynq boards yet) including generating boot.bin and pmu/bl31

https://github.com/rsarwar87/koheron-sdk/tree/xilinx_2020

would it be useful to move everything to 2020?

alternatively, if there was a list of changes that were made to the kernel/uboot source, I could attempt to do so as i can test them on microzed/alpha250 devices that i have access to.

Br Rashed

[jeanminet]

Hi Rashed, We were just discussing this today! I'm starting to work on a 2020.1 branch (#524), with the goal of merging it into master when it will be ready. You are much more advanced than we are... How could we incorporate your work into the 2020.1 branch ? Thanks! Jean

[rsarwar87]

Hi Jean

yes, it would be a very useful thing to have, as xilinx seems to have stopped support for 2017 versions. and the newer 2018 are too buggy to me. 2019.1 works well though. but that means i have use two different versions of vivado/sdk to have full device support for building boot.bin. since xilinx is moving away from most things not xsa and vitis, the sdk needs to be moved forward sooner rather than later. I have been testing 2020.1 for a while now and it seems stable enough for production work.

I could help with the initial support, i.e. bringing your 2020.1 branch up to the same stage as mine and make a pull request.

however, sometimes i tend to do things slightly differently. I would recommend having a look at the diff from the 2020.1 branch in my fork of the repo: https://github.com/rsarwar87/koheron-sdk/commit/1fd0ffd126fc440fba26e1c1e931b8d2dd50bd1a https://github.com/rsarwar87/koheron-sdk/commit/59bc3dfdfb64028d8ee3a0250fd20c78e88bcfad

there is no issue with the bitstream generation.

One major change is that there is no xdevcfg, so bitstream has to be written to configfs through overlay which also changes the clocks. alternatively, you could write to

mkdir -p /lib/firmware

cp /media/design_1_wrapper.bit.bin /lib/firmware/

echo design_1_wrapper.bit.bin > /sys/class/fpga_manager/fpga0/firmware

and then fix the clocks in the present way. in both cases fpga.mk needs a bit of modification to include the building of *.bit.bin file. I made those changes on Feb 6/see below. for overlays to work, i had to modify the kernel source a little, as i could not find the right flag to do it. here

on the OS side, the main thing is HSI has been removed in favour of XSCT, so those tcl scripts have to be rerouted to xsct. and unfortunately, XSCT does not take arguments, so I had to improvise a dirty workaround a little. That is all that is needed. other major change is the way boot.bin is generated.

The commit in the link above builds on the extension that I made on Feb 6/18/22 of this year to bring ultrascale support to the SDK, based on your merge for v0.20

you can find more of the extension on here -- commits made on Feb 6/18. some of those commits related to ubuntu 18.04.1 support. Some of those changes are no longer relevant with 2020.1.

if you are happy to accept these changes, I be happy to reorganize my changes and add them to your 2020.1 branch and make a pull request. we can then take it from there and test the zynq devices too. for this, we would also need the kernel source and the uboot sources in your fork needs to be moved forward to 2020.1.

EDIT: added new link to commits

[jeanminet]

Thanks a lot!

We don't plan to use the forks of the kernel and the u-boot source anymore. We started doing it almost two years ago but never merged it on to master...

On my first try with 2020.1 today, bitstream generation was smooth and but it then stopped on HSI. I did not realized that it was replaced by XSCT so I sourced the 2019.1 settings to use HSI...

I'm going back to 2019.1 for the moment and I am having a problem with the device tree compilation. Could you tell me why you use dtc-1.5.0 instead of the one provided by Xilinx?

[rsarwar87]

No problem - anything i can do to help. I have been telling people to stop using vitis and use this unless they want to use HLS/OpenCL

The device tree compilation issue probably has to do with mismatch in version 2020.1 and 2019.1. 2020.1 donot generate a hwdef but rather a xsa file.

I have some board that uses petalinux 2019.2; built externally to the sdk. i did not want to download the whole linux source to compile the overlay, so i downloaded the dtc-1.5.0.

This is another point that you may wish to consider - making the boot.bin using petalinux rather than downloading sources directly.

The other thing to consider would be to export the linux kernel header/source (for the dev image) and kernel modules (for both production and dev) and place them in the image itself. i.e. make modules_install INSTALL_MOD_PATH=$root_dir and make headers_install INSTALL_HD_PATH=$root_dir/usr/

it may be necessary in certain cases - e.g. I was working on a porting minized to your sdk, and had to export some modules that only work when placed in the rootfs

[rsarwar87]

I was looking at compiling for alpha250-4, and these are the changes that i needed to do

fix for floating inputs: https://github.com/rsarwar87/koheron-sdk/commit/ab9cce6550029cc4a119fc03f3244bfcd96a8d62 https://github.com/rsarwar87/koheron-sdk/commit/cb9f7df910cb98b04afbde21e1725aa07269971a

These fixes may not be needed if you are not using OCC. but i use occ, so i needed to make these.

to compile with newer dtc, there are two options,

1. you could modify makefiles in the linux-source to compile them, i.e. what i am doing for my enclustra boards and copy the dts from there: https://github.com/rsarwar87/koheron-sdk/tree/xilinx_2020/boards/mercury_zx1_pe1/patches/linux/arch/arm/boot/dts
2. update your os/os.mk to use the right modes: https://github.com/rsarwar87/koheron-sdk/commit/a733a9edb45f5236e011304da8b05fbed103e5de https://github.com/rsarwar87/koheron-sdk/commit/8c1dcd4d8873a0b0bee6487cbb045e64812360ba https://github.com/rsarwar87/koheron-sdk/commit/932166d9007e4de2e08ef04fa863fba8cae6f3b6 -- this one will give allow you to use overlays, otherwise should not be needed.

i think i made some changes to the alpha250-4 which should give it support for 2020.1 https://github.com/rsarwar87/koheron-sdk/commit/a733a9edb45f5236e011304da8b05fbed103e5de however, i am yet to test it on the board. I am not sure if I managed to get the ethernet devicetree node correctly. I am assuming it is similar to what enclustra uses, in which case this should work. otherwise, you may need to suggest to me what the dts should be.

[jeanminet]

Thanks! I went through all your commits on your branch xilinx_2020 and cherry picked the ones related to the SDK (I excluded the commits related to boards/ and examples/ for now): #525

[rsarwar87]

Great, do let me know if you tested on the board. I am not confident if the uboot is being compiled correctly for the alpha250. I have little understanding of uboot build infrastructure.

I will get back to the 2020.1 branch tomorrow to do more testing.

[jeanminet]

Hi Rashed,

I'm stuck at this stage:

source ./fpga/vivado/hwdef.tcl
# set xpr_filename [lindex $argv 0]
# set hwdef_filename [lindex $argv 1]
# set nCPU [lindex $argv 2]
# open_project $xpr_filename
Scanning sources...
Finished scanning sources
INFO: [IP_Flow 19-234] Refreshing IP repositories
INFO: [IP_Flow 19-1700] Loaded user IP repository '/home/jean/github/koheron-sdk/tmp/examples/alpha250/adc-dac-bram/cores'.
INFO: [IP_Flow 19-2313] Loaded Vivado IP repository '/tools/Xilinx/Vivado/2020.1/data/ip'.
# if {[get_property PROGRESS [get_runs synth_1]] != "100%"} {
#   launch_runs synth_1 --jobs $nCPU
#   wait_on_run synth_1
# }
# write_hw_platform -fixed -include_bit -force -file $hwdef_filename
INFO: [Vivado 12-4895] Creating Hardware Platform: tmp/examples/alpha250/adc-dac-bram//fpga/adc-dac-bram.xsa ...
INFO: [Hsi 55-2053] elapsed time for repository (/tools/Xilinx/Vivado/2020.1/data/embeddedsw) loading 0 seconds
ERROR: [Common 17-70] Application Exception: Unable to get BIT file from implementation run. Please ensure implementation has been run all the way through Bitstream generation. Aborting write_hw_platform..
INFO: [Common 17-206] Exiting Vivado at Mon Jun 29 20:16:34 2020...
fpga/fpga.mk:66: recipe for target 'tmp/examples/alpha250/adc-dac-bram//fpga/adc-dac-bram.xsa' failed
make: *** [tmp/examples/alpha250/adc-dac-bram//fpga/adc-dac-bram.xsa] Error 1

Do you have any idea of what is going on?

[rsarwar87]

Hi. yes. I forgot about this. It occurs because the new xsa scheme needs STATUS of impl_1 as "write_bitstream Complete!". Obviously, we have already generated the bitstream, but that flag never gets update. Maybe write_hw_platform does not know the location of the bitstream aswell

This is the workaround i have for now. I have not pushed it yet, as I am hoping to find a more elegant solution of a way to update the STATUS flag without having to reproduce the bitstream.

diff --git a/fpga/vivado/hwdef.tcl b/fpga/vivado/hwdef.tcl
index 2079c887..7fb1f11c 100644
--- a/fpga/vivado/hwdef.tcl
+++ b/fpga/vivado/hwdef.tcl
@@ -8,6 +8,10 @@ if {[get_property PROGRESS [get_runs synth_1]] != "100%"} {
   launch_runs synth_1 --jobs $nCPU
   wait_on_run synth_1
 }
+if {[get_property STATUS [get_runs impl_1]] != "write_bitstream Complete!"} {
+  launch_runs impl_1 -to_step write_bitstream 
+  wait_on_run impl_1
+}
 
 write_hw_platform -fixed -include_bit -force -file $hwdef_filename 

[jeanminet]

Thanks! At this stage (d36fb8a), I'm able to compile the image for the ALPHA250 (adc-dac-bram) but the LED FPGA_DONE stays off and I don't have access to the console. I will investigate...

[rsarwar87]

Hi

I see that you made some new commits after your last post. Can i assume that the boot issue has been fixed?

Best Regards Rashed

[jeanminet]

At this stage (267160a), the ALPHA250 fft design works at expected. I now plan to:

• Load the bitstream with FPGA manager from the server instead of bash
• Program the clock (fclk0...) from the server
• Upgrade the Ubuntu version from 18.04 to 20.04

[rsarwar87]

Managed to get Microzed booted. https://github.com/rsarwar87/koheron-sdk/commit/bfc458e136a044dfc834b3a71e067ed7203317eb https://github.com/rsarwar87/koheron-sdk/commit/03e19acb2b4c0ef0a72853d9cb7e11e08a0d3441

Was able to load bitstream using overlays as well.

needed to fix a typo to get it to work https://github.com/rsarwar87/koheron-sdk/commit/af22c7ef43a1cf3ebbf11bceffc9bc40ec0edc13

there are a few error messages, I will investigate the implications of it, but i suspect it is not fatal

[   10.345621] fpga_manager fpga0: writing fel-unit.bit.bin to Xilinx Zynq FPGA Manager
[   10.359485] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
[   10.368764] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/overlay0
[   10.377335] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/overlay1
[   10.385888] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/clocking0
[   10.394533] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/clocking1
[   10.403176] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/clocking2
[   10.411790] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/overlay2
[   10.420346] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/ctl_axi_ctl_register_0
[   10.430112] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/fel_unit_0
[   10.438863] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/misc_clk_0
[   10.447607] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/psa_axis_fifo
[   10.456598] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/sts_axi_sts_register_0
[   10.663611] macb e000b000.ethernet eth0: link up (1000/Full)
[   10.663638] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

EDIT: updated links

[pavel-demin]

For information. Bringing xdevcfg back is relatively easy. I'm using it with Linux 4.19 and 5.4 without any problem. You'll need to copy xilinx_devcfg.c to drivers/char/xilinx_devcfg.c and apply this patch.

It occurs because the new xsa scheme needs STATUS of impl_1 as "write_bitstream Complete!".

Out of curiosity. Why do you need bitstream for xsa?

I thought that xsa doesn't require anything and the following simplified version of hwdef.tcl works for my ZYNQ projects:

set project_name [lindex $argv 0]

open_project tmp/$project_name.xpr

write_hw_platform -fixed -force -file tmp/$project_name.xsa

close_project

[rsarwar87]

@pavel-demin its because of the include_bitstream flag. I guess one could remove that flag as we dont really use it.

regarding the patch for devcfg, one would need to recreate the patch every time with a kernel update. besides, fpga manager is stable since 2019.2 to be using in production. also you can write fpga0/firmware to get the same result without needing to patch the kernel echo design_1_wrapper.bit.bin > /sys/class/fpga_manager/fpga0/firmware

@jeanminet I tested out ubuntu20. image preparation fails as i cannot seem to find python2-pip or wheel. i tried the get-pip.py, that failed too for some reason. might be worth upgrading it to python3?

[pavel-demin]

regarding the patch for devcfg, one would need to recreate the patch every time with a kernel update.

There isn't much to recreate. The driver itself doesn't change because Xilinx has stopped updating it. The patches for Kconfig and Makefile are trivial. Anyway, I'm just sharing information in case anyone is interested in this type of solution.

[pavel-demin]

BTW. If you're using the Red Pitaya boards, the PHY driver requires patching because the default skew timings don't work correctly with the Red Pitaya boards. More details can be found at the following link:

https://forum.redpitaya.com/viewtopic.php?f=9&t=1880

Here is a link to a patch that I use to correctly set the PHY skew timings:

https://github.com/pavel-demin/red-pitaya-notes/blob/master/patches/linux-5.4.patch#L195-L237

[rsarwar87]

@pavel-demin I did consider it after finding your patch for 4.14 when i was struggling with xdev last year. :)

It is a philosophical debate for me. I did a similar patch for enclustra phys working on the ZX and UX boards, i.e. ported 4.9 macb drivers to 4.19/5.4, and my experience is that do such an extensive hack seems to deter people from wanting to use it in production machines. I was asked to do make detailed testing of the driver performance and what not

And having overlays have its own advantages as we can load the xilinx driver for a given device and use it. it helps with quick prototyping before streamlining the drivers for real-time applications using the sdk framework. :)

I have red-pitaya support next on my list, those patches would really be helpful :)

[rsarwar87]

@jeanminet

this patch will get overlay and non-xdevcgf flashing capabilities in the koheron-fpga-manager.

Not entirely sure how to minupulate clocks in these kernels. but overlays are able to take care of it.

https://gist.github.com/rsarwar87/2832247f6982fe36e8f31bda30af052a

[jeanminet]

@jeanminet

this patch will get overlay and non-xdevcgf flashing capabilities in the koheron-fpga-manager.

Not entirely sure how to minupulate clocks in these kernels. but overlays are able to take care of it.

https://gist.github.com/rsarwar87/2832247f6982fe36e8f31bda30af052a

Thanks a lot for this! I applied your patch: a00486772fe020d3a9dc

I tried changing the clock fclk0 in pl.dtsi but it does not seem to have any effect when I load the bitstream :( This driver may be a solution: https://github.com/ikwzm/fclkcfg

[rsarwar87]

Interesting. Did you check if there is an entry for Fpga manager updating the device tree from dmesg?

I check it to be working with zynqmp, will check on zynq later today.

This post does seem to confirm it does work from overlays https://forums.xilinx.com/t5/Embedded-Linux/Setting-PL-clock-frequency-in-Petalinux-2018-3/td-p/958482

Also I am not sure if changing the pl.dtsi without actually changing the bitstreams works.

[jeanminet]

Currently, the bitstream name is not defined in the pl.dtsi (firmware-name should be xxxx.bit.bin)

/dts-v1/;
/plugin/;
/ {
	fragment@0 {
		target = <&fpga_full>;
		overlay0: __overlay__ {
			#address-cells = <1>;
			#size-cells = <1>;
			firmware-name = ".bin";
		};
	};
.....

The bitstream is thus not loaded when echoing pl.dtbo to /configs/devicetree/overlays/full/path

[rsarwar87]

Looking at this patch: https://github.com/Koheron/koheron-sdk/commit/f82d50651cf0bb639c49560341f87294dafa0787 That makes sense, It does not know the name of the bit.bin file as we did not include it in the xsa. I have not downloaded your latest branch, but can i ask you to undo the patch and try again please

Sorry i forgot about this when i made the post suggesting we can ignore the flag. I think i had this issue before i fixed it by adding that flag.

[rsarwar87]

Alternatively, you can have the makefile issue a sed command to replace .bin to $project_name.bit.bin

Note that i did not commit that changes in the koheron fpga_manager.hpp as I have not tested it for all failure modes.

[jeanminet]

Thanks! I'll use the sed command (that you have already written!).

Regarding the overlay, I have renamed overlay.dtb to pl.dtbo to follow the Xilinx convention: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841645/Solution+Zynq+PL+Programming+With+FPGA+Manager#Test%20Procedure

I think that what is related to the overlay should be in /fpga and /examples (not in /os and /board). What do you think?

[rsarwar87]

I was thinking the same last night. - that there is no sense for overlay.patch under the board directory. and so if we should have one, it should be in the example directory. also need to have a flag that checks if the overlay.patch file exists. But, with the koheron model, I can hardly see a use for having an overlay.patch, unless one his to write a kernel module for their custom IP, which needs to be loaded by the pl.dtbo.

If you wish to move the makefiles to the fpga directory, i think you need still load the os.mk file as it requires the download links of the Xilinx devicetree repo.

The other thing that you may want to have an input on is: overlay and devicetree do not have to be generated in two different directories, i.e. /tmp//os/devicetree and /tmp//os/overlay are the same set of files. The only reason for doing so when I did it is because the script that generates the devicetree.patch gets confused when both pl.dtsi and pcw.dtsi are changed and lumps them all in the same file. Not sure how to address it though.

pl.dtbo should perfect. I am horrible when it comes to naming things - as you may have already realized. :)

[jeanminet]

overlay and devicetree do not have to be generated in two different directories, i.e. /tmp//os/devicetree and /tmp//os/overlay are the same set of files.

At least it works for now... At this stage (1b802fdd3d134b), loading and reloading the bitstream works through overlay. See dmesg ouput below:

[   10.234651] fpga_manager fpga0: writing fft.bit.bin to Xilinx Zynq FPGA Manager
[   10.272202] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
[   10.281557] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/overlay0
[   10.290284] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/overlay1
[   10.298849] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/clocking0
[   10.307489] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/clocking1
[   10.316130] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/overlay2
[   10.324683] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/ctl_axi_ctl_register_0
[   10.334448] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/misc_clk_0
[   10.343172] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/psd_axi_bram_ctrl_demod
[   10.353031] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/psd_bram_axi_bram_ctrl_psd
[   10.363153] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/sts_axi_sts_register_0
[   10.372927] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/xadc_wiz_0
[   10.427107] xadc 43c00000.xadc_wiz: IRQ index 0 not found

However, fclk0 does not seem to be updated when I change the frequency in pl.dtsi:

diff -rupN overlay.orig/pl.dtsi overlay/pl.dtsi
--- overlay.orig/pl.dtsi	2020-07-03 15:49:15.733347068 +0200
+++ overlay/pl.dtsi	2020-07-03 15:49:37.326092622 +0200
@@ -21,7 +21,7 @@
 		overlay1: __overlay__ {
 			clocking0: clocking0 {
 				#clock-cells = <0>;
-				assigned-clock-rates = <200000000>;
+				assigned-clock-rates = <20000000>;
 				assigned-clocks = <&clkc 15>;
 				clock-output-names = "fabric_clk";
 				clocks = <&clkc 15>;

[rsarwar87]

okay, im with a microzed, ill test and report back in a bit.

how did you check the clock speed?

i got completely different designs on microzed, one with fclk0 and 1 and the other has fclk0,1 and 2 enabled. all at different clockspeeds.

I can see them accordingly under /sys/firmware/devicetree/base/amba as cloocking# i.e. number of clocks seen under the amba directory changes with different overlays

cant seem to read the clockspeeds or to set them yet, ill keep digging, maybe add a frequency counter.

I think you would need a matching bit.bin with the modified pl.dtsi. I could be wrong

[rsarwar87]

maybe you can see if you can read /sys/firmware/devicetree/base/amba/clocking0/assigned-clock-rates

maybe it is in binary. odd

EDIT: not entirely sure what that means

root@koheron:/sys/firmware/devicetree/base/amba/clocking0# hexdump assigned-clock-rates 
0000000 f505 00e1                              
0000004

[jeanminet]

I'm only relying on the speed of the web interface (which reads bram from fclk0)...

fclk0 set to 200 MHz (through pl.dtsi):

root@koheron:/sys/firmware/devicetree/base/amba/clocking0# hexdump assigned-clock-rates 
0000000 eb0b 00c2                              
0000004

fclk0 set to 100 MHz

root@koheron:/sys/firmware/devicetree/base/amba/clocking0# hexdump assigned-clock-rates 
0000000 0100 a086                              
0000004