Context utilization UI & UX (8k, 4k, ..)

[enricoros]

Measure the token usage of a chat, to know when we're about to hit the limit, and offer ways to deal with it.

UX could be a progress bar with utilization and remaining capacity when composing a new message. A red bar would hint at removing some older chat messages before being able to compose a new message. An alternative would be to automatically prune messages from the start of the conversation (but keeping the system message).

Tech note: context windows vary by Model - GPT-4 8k having the largest until the 32k comes along.

Note: originally mentioned by by @bytesuji in https://github.com/enricoros/nextjs-chatgpt-app/issues/2#issuecomment-1475641938_