TK-Master
TK-Master
please accept this pr @abetlen
Actually.. I found an issue with this method.. this will only cancel after a token is generated but if the llm is slow or gets stuck processing the prompt, this...
I'm coming back to this because I need to figure out a better method to interrupt the generation programmatically.. For a console-based scenario it's pretty easy in python, all I...
I tested the code and I got no errors on windows, seems like an issue on mac