Improve experience for folks using a largish model where you get say 1 or less tokens per second

[TyDunn]

Validations

• [X] I believe this is a way to improve. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that requests the same enhancement

Problem

I'm playing with continuedev currently (https://marketplace.visualstudio.com/items?itemName=Continue.continue). Not super happy with it, especially while trying slower local models.

1. Visibility to issues: I don't know if something is moving, or timed-out. Often I have to restart the extension host. I suspect some stuff times out on the extension side, but I don't have the time to debug that.
2. Visibility to the comm: I see messages are being exchanged with the AI (on the command line.) A "log" window/pane may solve this.
3. Making it edit things inline: again, sometimes it just stops.
4. I couldn't see any way to "cancel" while it's editing my source files. I can see that it's clearly going towards a wrong direction, yet I can't do much about it.
5. Accept/Reject buttons sometimes randomly do not work. Closing and re-opening the source file does not change this.

I'm basically back to using text-gen UI for coding tasks, and copy pasting.

I'd recommend trying it out with a largish model where you get say 1 or less tokens per second and you'll probably see these issues too.

Reported by a user on r/LocalLlama here.

Solution

Try out Continue with a largish model where you get say 1 or less tokens per second (better: using VS Code and TextGenWebUI on Windows too) and address the issues listed above that pop up

[sestinj]

Open questions:

• Haven't had luck pinning this down yet: Does accept/reject buttons not working mean they don't show up, that clicking them doesn't affect anything, or that clicking them incorrectly modifies the file?
• In general, I think we hadn't considered the "let a large model do it's thing while you're cooking dinner, etc..." technique but it makes a ton of sense. If there are other niceties that would make this easier, curious to hear about them

[sestinj]

Improvements made so far:

• I added an output channel that will show you every prompt/completion to/from the LLM

• Error messages for fetch failures will now actually tell you what's going on, including if it's a TimeoutError

• There was no way to cancel when streaming inline edits from "ctrl+shift+L". I added a new button that will now show up in the editor title bar. For '/edit' there's the usual button at the bottom of the chat window to cancel.

• I made an OpenAI-compatible server that can be used to double-check Continue's timeout behavior. It seemed not to timeout for me up to an hour, but in case you want to manually set a longer timeout we have a timeout parameter in requestOptions, documented here.

• It's likely that some of the streaming issues you were seeing were related to a background request to generate a title for the current session. To make sure this never again blocks the server I disabled it for any local model provider

• Previously the tutorial was only opened once upon initial download, so our later updates to modify shortcuts for Win/Linux wouldn't have taken effect. I added a button in the help center to re-open the tutorial, which will make sure that shortcuts are converted.