hls4ml
hls4ml copied to clipboard
fastmachinelearning
Cannot synthesizing large models
Hello. I am trying to use hls4ml to convert the codes into HLS and then synthesize it. Although the hls4ml works for small models, I get this error when I increase the model size for example for mnist:
Stop unrolling loop ........ because it may cause large runtime and excessive memory usage due to increase in code size. Please avoid unrolling the loop or form sub-functions for code in the loop body. ERROR: [HLS 200-70] Pre-synthesis failed.
I really appreciate if you help me what is the problem. Thanks
I ran into the same issue. My model still does not synthesize completely but I am not getting that error anymore. Try to change the Model Strategy in the config file or settings to 'Resource' and increase the ReuseFactor (I went up to 64 to bypass that error for my case)
Thank you for your response. Actually I solved the issue and passed the mentioned error. However, the synthesis fails again and shows this error: line 164: 494 Killed "$RDI_PROG" "$@". I am not sure if HLS4ML can handle big models or there is a problem with my codes. Do you have any idea about this error? Thank you very much
hls4ml (currently) stores all weights on-chip in order to minimize the latency of the designs. Naturally, this is limiting the size of the models and as you have observed it is not possible to put any model on the FPGA. How large is a "large model" is also not easy to answer, as it depends on the architecture of the model, the precision/quantization used and pruning (if any). I would suggest you try with the smaller models of the architecture that you want (you don't need to train them), just to get an estimate of the number of parameters that you can fit.
Thank you very much for your response. I will try to follow your instruction. This problem also happens when I try to follow the HLS4ML Jupiter notebooks in part 6 (CNN). After completing the notebook, I get this error:
/opt/Xilinx/Vivado/2019.2/bin/loader: line 280: 9050 Killed "$RDI_PROG" "$@" Synthesis report not found.
Can you please help me what should I do for this part and why I get this error even for the available CNN tutorial notebook (https://github.com/fastmachinelearning/hls4ml-tutorial). Thank you very much.
I ran into the same problem numerous times, check your systems memory usage during the HLS compilation, I think the error comes from Vivado HLS running out of RAM. I solved this by incresing the ReuseFactor of the Layer that was being converted when the error occured.
Alternatively you could try to use fewer parameters in that layer, use a system with more RAM or increse the size of the swap file, but that would probably also increase the runtime by an order of magitude. Alternatively you could try pruning but I have not tried that and do not know if that would fix it.
I have the same issue and will keep an eye on RAM as suggested. Additionally, is there any report on model size/complexity feasibility using hls4ml with certain reuse factor and for example a KU115 FPGA?
