llama.cpp icon indicating copy to clipboard operation
llama.cpp copied to clipboard

Published 20 hours ago • verified publisher icon ggml-org

Don't force immediate interactive without `-i`

Open tjohnman opened this issue 2 years ago • 3 comments

Sometimes we might want to use a reverse prompt but we want to let the model generate tokens right after the initial prompt. So we don't force user input mode if the -i flag wasn't specified and instead let it run until we encounter the reverse prompt.

This gives use some more flexibility, since it doesn't force the user to enter a newline if they want to let the model generate text right after the initial prompt and only be asked for input if the reverse prompt is encountered.

tjohnman avatar Mar 21 '23 14:03 tjohnman

I think it's good to not force interactive mode immediately (in fact that was how it worked when I first made the patch, but the logic seems to have changed at some point), but in this combination the flags seem to be rendered a bit misleading.

What I conceived of in the beginning:

  1. No -i or -r PROMPT: Classical, non-interactive generate-only mode.
  2. -i: Interactive mode, in which the user may add additional input during generation by pressing Ctrl+C to seize control.
  3. --interactive-first: As -i, plus prompt for user input immediately after the initial prompt is emitted.
  4. -r PROMPT: As -i, plus look for reverse prompt PROMPT, so control passes to the user whenever it is encountered.

The scenario you are proposing:

  1. No -i or -r PROMPT: Classical, non-interactive generate-only mode.
  2. Not available: interactive mode, where further user input is prompted after Ctrl+C only.
  3. -i: Interactive mode, plus prompt for user input immediately after the initial prompt is emitted.
  4. -r PROMPT: Interactive mode, plus look for reverse prompt PROMPT.

I think (2) is a valid use case, and moreover it is confusing that the "interactive mode" flag -i actually takes you into the "interactive + input in the beginning" mode while in order to get into any sort of interactive mode where you do not want to start out submitting input, you have to specify a reverse prompt with -r and not submit -i (and the natural UX for attaining 2 is -r RANDOM_STRING_THAT_NEVER_WILL_COME_UP).

Rather than loading initially_interacting from params.is_interactive, I would therefore suggest (re)introducing a dedicated parameter corresponding to it.

blackhole89 avatar Mar 21 '23 16:03 blackhole89

@blackhole89 I agree 100% with you that the first scenario is the most intuitive and useful (I'll do the changes). I did not remove --interactive-first (but I do remember seeing it in a previous build; no idea what happened to it).

tjohnman avatar Mar 21 '23 16:03 tjohnman

@tjohnman Thanks! Wasn't meaning to imply you had anything to do with the removal - development has been moving quickly and chaotically, it probably just fell on the wayside in some refactoring along the way.

blackhole89 avatar Mar 21 '23 16:03 blackhole89

this kind of broke instruction mode. this change needs to be only for --interactive not for --instruct

Green-Sky avatar Mar 22 '23 20:03 Green-Sky