Can we make it run on other models?

[YLSnowy]

Can we make it run on other models? Can the offload code be made public?

[hodlen]

Thank you for your interest! We are actively working to expand our range of supported models. However, there are certain limitations, as detailed in our FAQs. Please keep an eye out for updates on new model integrations!

Regarding the "offload code," could you please clarify which aspect you're referring to? If it's about neuron offloading, our implementation is fully open-sourced and available in this repository for your review and use.

[YLSnowy]

Thank you for your answering. So how to profile for a model ? I haven't seen source code about it.

[hodlen]

We are still organizing the code for profiling and training the predictor and will release them once they are ready. Please stay tuned for our progess in #93.

As for now, you can check the reference inplementaion in Deja Vu and refer to these related issues: #84, #54 .

[drewskidang]

question we can covert our own fine tunned llama models??

[hodlen]

question we can covert our own fine tunned llama models??

Please kindly refer to #17, #34, #56, #82, #94 and more discussion in previous issues.

[linkerlin]

I believe that a statistical method could be employed to set all outputs of non-ReLU activation functions that are below, for instance, the 30th percentile to zero, in a similar manner to obtain sparsity guarantees akin to those provided by ReLU.