xls icon indicating copy to clipboard operation
xls copied to clipboard

modules/zstd: Rework ZSTD Decoder

Open lpawelcz opened this issue 4 months ago • 1 comments

This PR reworks the existing ZSTD Decoder to memory-based approach instead of relying on stream-based architecture.

The PR relies on peripherals (MemReader and MemWriter) for accessing external memory through AXI interface that are introduced in: https://github.com/google/xls/pull/1613. It is also based on https://github.com/google/xls/pull/1616.

The Decoder is able to decompress ZSTD frames that consist of RAW and RLE blocks. This is tested in a verilog simulation with a cocotb testbench.

The changes in this PR include:

  • Rewriting ZstdDecoder proc to work with memory-based approach
  • Adding verilog-level tests of the ZstdDecoder
    • Introducing python library for generating ZSTD frames with decodecorpus
    • Adding python bindings for ZSTD reference library (zstandard) used for crosschecking decoding results in the simulation
    • Adding cocotb testbench that generates multiple ZSTD frames with RAW and RLE blocks and decodes those with simulated ZstdDecoder
  • Reworking FrameHeaderDecoder, BlockHeaderDecoder, Raw- and RleBlockDecoders to use memory-based approach instead of streaming input data from the input of the decoder
  • Adding CsrConfig and AxiCsrAccessor procs that implement the internal registers of the ZstdDecoder and provide external (AXI) and internal (native) facing interfaces for accessing those
  • Simplifying:
    • RleBlockDecoder proc
    • BlockHeader library
  • Removing components not used in memory-based approach:
    • magic number functions
    • BlockDecoder proc
    • DecDemux proc
  • Added common_codegen_args list to the BUILD file

This PR is currently a WIP. It is opened to showcase the work done and to create a space for discussion on certain topics. The following list enumerates task that are currently being addressed and which are required for this PR to be completed before it is marked as ready for review.

Next steps:

  • [x] Update documentation - describe the new architecture of the decoder and how to interact with the IP from the software (CSRs)
  • [x] Add DSLX tests for the ZstdDecoder proc - create facilities for preparing test data and write DSLX tests
  • [x] Improve error handling - write error codes to the Status CSR
  • [ ] Improve verilog tests - reorganize testbench to handle decoding of multiple ZSTD frames in one test case
  • [ ] Change the output format from stream-based to memory-based (write decoded frame to memory)
  • [ ] Organize structs and new type aliases (cleanup)
  • [x] Remove a workaround (https://github.com/google/xls/pull/1654/commits/85f1f80c7ddcf7016a282087bb342f16d313b1d6) for type check failure during IR generation for the ZstdDecoder proc (https://github.com/google/xls/issues/1655)

CC @proppy

lpawelcz avatar Oct 09 '24 15:10 lpawelcz