xtrx_julia

xtrx_julia copied to clipboard

                           _  ___________  _  __       __     ___
                          | |/_/_  __/ _ \| |/_/_____ / /_ __/ (_)__ _
                         _>  <  / / / , _/>  </___/ // / // / / / _ `/
                        /_/|_|_/_/ /_/|_/_/|_|    \___/\_,_/_/_/\_,_/
                         / ___/__  __ _  ___  __ __/ /___ _____  ___ _
                        / /__/ _ \/  ' \/ _ \/ // / __/ // / _ \/ _ `/
                        \___/\___/_/_/_/ .__/\_,_/\__/\_,_/_//_/\_, /
                                      /_/                      /___/
                            XTRX LiteX/LitePCIe based FPGA design
                                  for Julia Computing.

[> Intro

This project aims to recreate a FPGA design for the XTRX board with LiteX/LitePCIe:

This Julia Computing remote contains support for GPU P2P operations.

[> GPU Setup

For P2P operation the open source Nvidia drivers are required. Note that we currently carry our own patch for resizing the addressable memory space, and until that is merged, the easiest thing to do is to build our fork of the driver, which can be done via make -C software nvidia-driver.

[> Getting started

[> Installing LiteX:

LiteX can be installed by following the installation instructions from the LiteX Wiki: https://github.com/enjoy-digital/litex/wiki/Installation

Be sure to use the latest master branch of the LiteX sources, as none of the currently-available tags include some of the changes we require.

[> Installing the RISC-V toolchain for the Soft-CPU:

To get and install a RISC-V toolchain, please install it manually of follow the LiteX's wiki: https://github.com/enjoy-digital/litex/wiki/Installation:

./litex_setup.py --gcc=riscv

[> Build and Test the design

Build the design and flash it to the board:

./fairwaves_xtrx.py --build --flash

Build the Linux kernel driver and load it. Note that by default, the current live kernel will be built against, but you can cross-compile for a target kernel version by setting USE_LIVE_KERNEL=false.

make -C software litepcie-kernel-module
sudo software/litepcie-kernel-module/init.sh

Note that if a thunderbolt carrier is in use, it may be necessary rescan the pci bus:

sudo bash -c 'echo "1" > /sys/bus/pci/rescan'

Build the Linux user-space utilities and test them:

make -C software litepcie-user-library -j$(nproc)
cd build/litepcie-user-library
./litepcie_util info
./litepcie_util scratch_test
./litepcie_util dma_test

If anything goes wrong, reset the device with:

sudo bash -c 'echo "1" > /sys/bus/pci/devices/0000\:02\:00.0/reset'

[> User-space software

To interface with the LMS7002M chip, we provide a SoapySDR driver. This driver is in turn built on top of MyriadRF's LMS7002M driver library, which is downloaded and installed automatically when you compile the SoapySDR driver:

make -C software soapysdr-xtrx -j$(nproc)
export SOAPY_SDR_PLUGIN_PATH="$(make -sC software print-soapysdr-plugin-path)"

The above snippet sets SOAPY_SDR_PLUGIN_PATH so that any SoapySDR application can find the XTRX driver. This can be used to execute the example Julia scripts in this repository:

make -C software SoapySDR.jl

cd software/scripts
julia --project -e 'using Pkg; Pkg.instantiate()'
julia --project test_pattern.jl

NOTE: sometimes, the SoapySDR driver doesn't properly initialize the SDR (indicated by test_pattern.jl reading only zeros, or test_loopback.jl only reporting under/overflows). This can be worked around by launching litepcie_test (e.g. litepcie_test record /dev/null 1024). Afterwards, the chip should be in a good state.

There is also a modified version of LimeSuite available that makes it possible to interactively configure the LMS7002M:

make -C software limesuite -j$(nproc)

You can then run it out of the build/soapysdr/bin directory. Note that we install to the soapysdr directory to simplify the path manipulation needed for SoapySDR module autodetection.

[> Development

Both using LimeSuite and SoapySDR it's possible to load LMS7002M register dumps, e.g., to test specific functionality:

• TX-RX FPGA internal loopback:

  LimeSuiteGUI (and open/load xtrx_dlb.ini)
  cd software/app
  make
  ./litex_xtrx_util lms_set_tx_rx_loopback 1
  ./litex_xtrx_util dma_test -e -w 12
  

• TX Pattern + LMS7002M loopback test:

  LimeSuiteGUI (and open/load xtrx_dlb.ini)
  cd software/app
  make
  ./litex_xtrx_util lms_set_tx_rx_loopback 0
  ./litex_xtrx_util lms_set_tx_pattern 1
  ../user/litepcie_test record dump.bin 0x100
  

• DMA+LMS7002 loopback test:

  LimeSuiteGUI (and open/load xtrx_dlb.ini)
  cd software/app
  make
  ./litex_xtrx_util lms_set_tx_rx_loopback 0
  ./litex_xtrx_util lms_set_tx_pattern 0
  ./litex_xtrx_util dma_test -e -w 12
  

To work with the embedded RISC-V CPU, the firmware (which is normally automatically compiled and integrated in the SoC during build) can be recompiled and reloaded with:

cd firmware
make
sudo litex_term /dev/ttyLXU0 --kernel=firmware.bin --safe

LiteScope:

litex_server --jtag --jtag-config=openocd_xc7_ft232.cfg
litescope_cli

GLScopeClient (WIP): https://github.com/juliacomputing/scopehal-apps/tree/sjk/xtrx1

[> Contact

E-mail: [email protected]