Project-Zipline
Project-Zipline copied to clipboard
Defines a lossless compressed data format that is independent of CPU type, operating system, file system, and character set, and is suitable for compression using the XP10 algorithm.
Project Zipline
Introduction
Project Zipline is a program to accelerate innovation in lossless compression. With this release Microsoft is making available:
- A new compression format called XP10 which is tailored for modern cloud datasets.
- All specifications for the entire pipeline.
- All RTL to support implementation for the entire pipeline.
- A testbench environment specifically developed for the VCS simulator.
Project Zipline is open-sourced under the MIT License, see the LICENSE file.
Specifications
The following Project Zipline specifications have been included under the “specs” directory in this repository:
- Project_Zipline_Compression_Specification
- Project_Zipline_Huffman_Encoder_Micro_Architecture_Specification
- Project_Zipline_Comp_lz77_Micro_Architecture_Specification
- Project_Zipline_Crypto_Engine_Micro_Architecture_Specification
- Project_Zipline_Decompression_Top_Micro_Architecture_Specification
- Project_Zipline_Keyblob_Micro_Architecture_Specification
- Project_Zipline_KME_Micro_Architecture_Specification
- Project_Zipline_Prefix_Attach_Micro_Architecture_Specification
- Project_Zipline_Prefix_Micro_Architecture_Specification
- Project_Zipline_SSB_Micro_Architecture_Specification
- Project_Zipline_Top_Micro_Architecture_Specification
Simulation Notes
Edit the file ./zipline.setup
for VCS, SYNTH, and VERDI environment settings.
Then source the file as follows:
source zipline.setup
Go to either the CCE_64, CDD_64, or KME run directory:
cd dv/CCE_64/run
cd dv/CDD_64/run
cd dv/KME/run
To build the simv executable:
make build_simv
Example for simulating a test:
make run_simv TESTNAME=xp10
Example for simulating a test with waves (.vpd):
make run_simv TESTNAME=xp10 WAVES=1
Example for simulating a test with Verdi waves (.fsdb):
make run_simv TESTNAME=xp10 VERDI_WAVES=1
All of the tests for the CCE_64, CDD_64, and KME can be run via a regress script in the associated "run" directory:
dv/CCE_64/run/regress
dv/CDD_64/run/regress
dv/KME/run/regress
All test files are located in the associated engine directory "tests". Each directory also contains a README file with a description of each test.
dv/CCE_64/tests
dv/CDD_64/tests
dv/KME/tests
A description of the programmable registers in the CCE/CDD/KME can be found in the register_doc directory. Please read register_doc/README for more details.
KME Notes
The KME RTL (rtl/cr_kme) has been modified to remove the following modules:
- AES engine in the random GUID generator
- SHA engines within the KDF function
- AES engine within the Key Decryption logic
With these reductions, the RTL only supports Key Types 1-6 without KDF and Key Type 0 without encryption/authentication
Additionally, the KME output has been reduced to support a single engine.
All modifications are identified with "KME_MODIFICATION_NOTE" comments in the RTL code.
Synthesis Notes
Note: For synthesis, please edit syn/Makefile to choose an LSF or dedicated machine with at least 16G of memory. Edit the contents of this line, before "dc_shell":
bsub -Is -q irv-cpx-M16 -R "rusage [mem=16000]" dc_shell -f syn_example.tcl | tee ./LOGS/syn_eample.tcl.
date '+%m%d.%H:%M'.log
To run synthesis for CDD
cd syn/cr_cddip
make syn
To run synthesis for CCE
cd syn/cr_cceip_64
make syn
To run synthesis for KME
cd syn/cr_kme
make syn