fmaclen
Feature Request: Restarting generation after device wakes up
The generation stops as soon as the device sleeps and does not restart after it wakes up. It would be great to have a auto resume generation feature when device wakes up.
I'm not entirely sure if this is technically possible, my gut feeling tells me it isn't possible but will need to investigate.
I had a simple algorithm in mind, not sure about the feasibility of this:
- For every result from the ollama generation api, save the last value as context.
- When the process wakes up (I didn't give this one a lot of thought, might not be possible), use the stored value to pass the context (context parameter in this doc).
- To make sure the stored context is not used for a fresh prompt, after every successful completion of generation (or forceful stopping of generation by the user), this saved context can be deleted.
What are your thoughts?
Right now, when you stop/cancel a completion at any time before it finishes, the incomplete completion is removed from the session and we reset the prompt so you can send it again.
Not gonna lie, I'm not a huge fan of this behavior. I'd personally prefer to keep the half-finished completion in the session and be able to edit it (or re-send my earlier prompt).
If we were to implement the change as I described it, where half-completions get saved, then if your device goes to sleep we could theoretically keep what was completed up to that point.
When your device wakes up we could allow you to ask the LLM to re-trigger the last sent prompt (maybe specifying the same seed) so you'll get the exact same answer you were getting before your device went to sleep. But in that case it will still be re-generating the completion from scratch.
I guess it would be possible to add a special type of prompt trigger where we ask the LLM to continue where it left of, so it doesn't have to write everything from scratch, but that's probably not a trivial change.