An Operating System Runtime for FPGA Applications

Copyright (c) 2018-2020, Wuklab, UCSD.

This is an operating system (OS) running inside FPGA providing familiar OS abstractions (e.g., mm, sched, net) for FPGA applications. This is an inmature early stage research prototype. Most of the code is written in Xilinx HLS. It has been tested on VCU118 and VCU108.

It's made public for educational purpose. Hope you find it useful.

Codebase Organization Principles

• Make each subfolder be an IP on its own. This means it should have its own scripts, rtl or hls code, and testbench.
• Hierarchy IPs will have dependency. We need to express the order explicitly in corresponding building scripts.
• Final large project should be expressed in a diagram or static RTL code that ultilizing small IP cores.
• Overall, this project will consist many small IPs, and they will be used internally to build large ones. Our goal is to be able to reuse IPs as much as possible, and be able to construct new IPs easily.

Codebase Directories

• Host Side:
  • host/: Host side network stack and DMA drivers
• FPGA Side:
  • alloc/: memory allocator
  • mm/: memory management
  • net/: network subsystem
  • kernel/: some PR related code
  • system/: Final big integrated systems
• Both world:
  • app/: Application can have both FPGA and Host code.
  • include/: header files used by both FPGA and host code. E.g., network headers.
    • include/fpga: headers used by HLS only
    • include/uapi: headers used by both HLS and host code
• Helpers
  • tools/: various helpers
  • scripts/: template script files
• Generated
  • generated_ip/: all generated IPs sleep here
  • generated_hls_project/: Vivado HLS project
  • generated_vivado_project/: Vivado project

Format:

• General
  • Make your readable for others.
  • Remove ending spaces and tabs.
  • Try to use Linux Kernel Coding Style.
• Vivado HLS
  • All Vivado HLS script use run_hls.tcl name.
• Verilog
• IP
  • Name IP consistenly. Use subsystem name as prefix. E.g., mm_axi_wrapper is the AXI wrapper IP under MM subsystem.

Tutorials

HOWTO Build

• Type make help to see detailed explanation.
• Type make at top-level will compile the whole project using default board.
  • Kernel IPs will be compiled first
  • Large system reside in system/ folder
• Type make at each subfolder will compile that folder only.

Workflow: you should compile the project when you first download the source code. All small, medium IPs, and big projects will be ready to use. After this, you can focus on the IPs you are building. Changes will be reflected automatically within Vivado.

Also, pay attention to which board you are targeting.

HOWTO use the Vivado script

The goal of using scripts are two-fold

• Make the building process easier
• Use Git to track source files only. Because by default Vivado will create bunch files, and all of them should not be tracked by Git. In this project, all Vivado project-mode related files are placed under generated_vivado_project/ folder, at the same folder where the corresponding script and source code are.

Those Vivado project-mode scripts are generated by Vivado itself. And those scripts can be used to rebuild the whole project AS IS.

Those are my unformal steps to create and hack those scripts:

Run:

• Run the script:
  • vivado -mode tcl -source run_vivado.tcl
    • A new generated_vivado_project/ will be created at the current folder
    • A new IP will be packaged and exported into generated_ip/ folder.
  • If you don't want to type it everytime, use a template Makefile.

Creation:

• Create the top-level script:
  • Use Vivado GUI to create a new project, add existing sources/tb/xdc, set part.
  • Use write_project_tcl to generate the script. You SHOULD let the new script know that the going-to-be-rebuilt project should be placed under under folder's generated_vivado_project. And you SHOULD name the script to run_vivado.tcl.
  • If there is no block diagram (BD) design within project, use: write_project_tcl -force -no_copy_sources -target_proj_dir ./generated_vivado_project ./run_vivado.tcl
  • If there are BD designs within project, use: write_project_tcl -force -target_proj_dir ./generated_vivado_project ./run_vivado.tcl

Hack:

• Add IP paths to top-level script:

  • This SHOULD be added to every script.
  • All our generated IPs are in generated_ip/ folder. This makes IP tracking easier. You can add few lines to the script so that Vivado knows where to look for new IPs. Check out scripts/template_vivado_ip.tcl for those lines.

• Package IP automatically:

  • This SHOULD be added to every script.
  • We want to let the script to package IP automatially. This can be achieved by adding few lines at the end of the script. Note that you need to specify the destination location, the name of the IP, and so on. All our generated IPs are in generated_ip/. For example, ipx::package_project -root_dir ../../generated_ip/mm_axi_wrapper -vendor wuklab -library user -taxonomy UserIP -import_files -set_current false -force
  • Package a BD as an IP: ipx::package_project -root_dir ../../generated_ip/mm_axi_rab_vcu118 -vendor wuklab -library user -taxonomy UserIP -module THE_BD_NAME -import_files

• Add more source/xdc/simulation files:

  • First solution, manually change the generated script. Try to find out how those existing files are added, then you can modify the script in a similar way. Tips: search top.v, add_files etc.
  • Second solution, add files via Vivado GUI, and then use write_project_tcl again to generate another script. Then check out the differences.

• Autogenerate BD wrapper

  • This MUST be added if the top-level design is a BD
  • The reason is: if the BD has AXI interface, this AXI will need have predefined frequency property (Does not mean it must run on this, just a property.). By default, it is 100 MHz. If an IP is generated from this BD, and used by a larger IP, this frequency cannot be changed, which is VERY unconvenient. However, if we create a wrapper for the BD, and package the whole project as an IP, then the frequency property can be automatically updated during IP integration.

• Other settings:

  • IMPORTANT!
  • Whatever changes you made via Vivado GUI (e.g., compiling order, xdc order, simulation top file), and if you want to save it for others to use, you need to use write_project_tcl to generate a new script and save the corresponding changes. Once you get familiar with Vivado commands, you should be able to do manually by changing script.

Caveats:

• Review the generated script
  • Check if any source files within Vivado auto-generated folder is used in the script. This happens if the write_project_tcl is used not correctly. Having auto-generated files essentially equals a chicken-and-egg issue.

For now, you can find example scripts at the mm/axi_wrapper/run_vivado.tcl, and mm/sys/run_vivado.tcl.

HOWTO handle IP version

Vivado only supports one IP version in each version. It's super annonying if we use generated script. Luckily, there is one way to workaround it. The following code replace the IP string with a variable. However, be careful if the updated IP has different ports.

set axis_data_fifo [get_ipdefs -filter NAME==axis_data_fifo]
Replace all `xilinx.com:ip:axis_data_fifo:1.1` with `$axis_data_fifo

HOWTO use the Vivado HLS script

The Vivado HLS script is relatively easier than Vivado script. You can find the template script in scripts/template_run_hls.tcl. You SHOULD customize the parts, files added, frequency, and so. If you want to automatically build multiple HLS under the same folder, use the scripts/template_generate_hls.sh

References

• UG1118 Vivado Creating and Packaging Custom IP