SamurAIGPT
Use any small models, like LaMini, Flan-T5-783M etc.
Hello, sorry for the fact I couldn't find the solution in issues and if the question is dumb, but looking for the answer and trying by myself didn't give the result.
Details: The problem is I have low-end PC which is capable of running Alpaca and Vicuna (both 7B), but quite slowly. On the other hand, trying different models I saw that models under 1B parameters run quite well. Mainly they are based on Flan-T5. They give good results as for my machine and quickly enough (about 3-5 tokens per second). Using it with text is another better point. For example, asking it "basing on this text, answer -..." I have almost perfect answer. But giving it text each time is bad practice as for me. I mean, time spend etc.
Short question: Is there any way to use this tool with any of these models?
- LaMini-Flan-T5-783M
- Flan-T5-Alpaca (770M or something)
- RWKV (under 1.5B)
- (any other good small models, under 1B parameters) If you give the detailed manual I will be very grateful! Solutions, other than privateGPT etc. are also welcome!
Thank you for understanding, answers and sorry for any inconvenience!
