Vito Plantamura
Vito Plantamura
hi, yes, OnnxStream (the library!) now supports dynamic shapes for inputs, via the m_support_dynamic_shapes option. This was done to be able to run TinyLlama and Mistral7B; the Stable Diffusion application...
hi, Thanks for opening this issue before sending any refactoring PR. I am against PR of this kind for two reasons: 1) the fact of having a single implementation file...
cool! Let me know about your progress here! thanks, Vito
hi, I don't think memory would be an issue, as SD 1.5 requires 1.1GB of RAM in OnnxStream in its least conservative configuration. I've tried the latest version of OnnxStream...
hi, can you try without the "--rpi" option? On the RPI 5 it shouldn't be necessary, but I haven't had the opportunity to test it. Can you then do a...
have you tried adding the "--not-tiled" option? I just remembered that LivingLinux managed to run OnnxStream on the RPI5, without any problems: https://www.youtube.com/watch?v=D0qG2OIpbUk Vito
As for the RPI5, it could be a XNNPACK problem. We could try with the same version of OnnxStream used by LivingLinux in his video, which is commit 580cd677310a70fe35c8aecbffbbaa012ae54855. The...
hi, the main reason is that OnnxStream allows to "stream" the parameters of a model. Therefore it allows to run very large models on devices with little RAM (however sacrificing...
Of course, make sure to specify the "--rpi-lowmem" option. Vito
hi, unfortunately not, for the moment at least... I have a series of optimizations in mind for OnnxStream in order to make it perform better on more powerful systems. It...