llama.cpp
llama.cpp copied to clipboard
ggml-org
Longer and infinite output
If we use -n 1000000 to have a very long output (for a story for example),
it stops generating quite fast, after around 30 lines, probably because of this line of code.
It would be nice if we could have longer outputs and also the possibility to have infinite output, stopping only on Ctrl-C.
We could maybe specify that -n 0 will trigger that infinite output mode.
That issue is a bit related to issue #23
Depending on how much memory you have you can increase the context size to get longer outputs. On a 64gb machine I was able to have a 12k context with the 7B model and 2k context with the 65B model. You can change it here
Huh... I thought the context size was determined when the model was trained due to the positional encoding used. (I am only a layman.) But https://github.com/ggerganov/llama.cpp/pull/78 is still useful for when you eventually hit the limit of your context, right?
When trying large contexts, I often encounter
aggml_new_tensor_impl: not enough space in the context's memory pool (needed 702840624, available 701883745)
I played around a bit with increasing ctx_size but that did not work, I suspect an underlying memory UB as the cause, as lldb seems to trap on some suspicious memory accesses
Depending on how much memory you have you can increase the context size to get longer outputs. On a 64gb machine I was able to have a 12k context with the 7B model and 2k context with the 65B model. You can change it here
Your link goes to this code snippet if (!llama_model_load(params.model, model, vocab, 512)) { // TODO: set context from user input ?? is this correct to change the context size?
ggml_new_tensor_impl: not enough space in the context's memory pool (needed 704905456, available 704155676) Assertion failed: false, file ggml.c, line 2516
@drewcrawford for me, that error doesn't appear to be context size related. I've run the same prompt at different context sizes and they all fail.
Typically if you get the not enough space in the context error you tried setting the context too large, though on the larger models I have had to tweak this line too. The math around memory allocation in this thing isn't perfectly scaling on the larger models and unfortunately my fork has substantially diverged from master and too lazy to work on merging.
llama_model_load: loading model from 'models/13B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx = 8192
llama_model_load: n_embd = 5120
llama_model_load: n_mult = 256
llama_model_load: n_head = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot = 128
llama_model_load: f16 = 2
llama_model_load: n_ff = 13824
llama_model_load: n_parts = 2
llama_model_load: ggml ctx size = 20562.56 MB
llama_model_load: memory_size = 12800.00 MB, n_mem = 327680
8192 context size on quantized 13B model
Yes, the math for computing necessary memory for the ggml buffers and graphs has to be updated.
What's the best way to enable infinite output? Can we just shift-out the old contexts in K and V tensors (along the n_ctx dim) when they are full, or is there a better approach?
Hmm, I think yes - we need to shift the KV cache. I haven't implemented this yet in any of the ggml examples.
And when the context is full - stop increasing n_past.
The 65B model sometimes crashes with 2k, but usually works ...
llama_model_load: loading model from './models/65B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx = 2048
llama_model_load: n_embd = 8192
llama_model_load: n_mult = 256
llama_model_load: n_head = 64
llama_model_load: n_layer = 80
llama_model_load: n_rot = 128
llama_model_load: f16 = 2
llama_model_load: n_ff = 22016
llama_model_load: n_parts = 8
llama_model_load: ggml ctx size = 49157.73 MB
llama_model_load: memory_size = 10240.00 MB, n_mem = 163840
The increased context doesn't increase memory use (still under 41GB) - until the context gets filled.
Yet, on the 7B model the calculation overflow at the start for 16k or greater. Usually, 15360 will work.
llama_model_load: ggml ctx size = 17592186044337.35 MB
https://github.com/eous/llama.cpp/commit/e0213e08c03a3ac72cdec4596b872073b51655aa here is some easy stuff I pulled out of my local hackery if anyone wants to play with it
Btw if anyone wants to slice/dice/refactor/cleanup/dissect/mixup/etc that changeset feel free, I don't need to be credited.
https://github.com/eous/llama.cpp/commit/e0213e08c03a3ac72cdec4596b872073b51655aa here is some easy stuff I pulled out of my local hackery if anyone wants to play with it
It's only been 4 days and this is apparently already a dead link. is that because this has already been added or?
https://github.com/ggerganov/llama.cpp/commit/e0213e08c03a3ac72cdec4596b872073b51655aa
I'm wondering what is happening when the context goes over the 2048 tokens limits (#274). The models seems to break down quickly, in unexpected ways.
As a non initiated to the field, I asked ChatGPT, but I'm not sure if I should believe it:
If the KV memory contains more contexts during inference than were used during training, it can lead to a phenomenon known as "model fragmentation". Model fragmentation occurs when the model becomes unable to retrieve useful information from the KV memory because the stored contexts are too diverse or unrelated to the current context. During training, the model learns to associate the input text with the corresponding output text by processing a set of training examples. If the KV memory contains more contexts during inference than were used during training, the model may encounter contexts that it has not seen before, which can lead to difficulties in retrieving relevant information from the memory. In other words, the model may not be able to generalize well to unseen contexts if the KV memory contains too many unrelated or diverse contexts.
From what I gather, an excessive number of retrieved values are summed up in the output, resulting in a signal loss. However, I find it surprising that the degradation is so fast and severe.
What are the strategies for rolling or pruning the contexts? As briefly discussed, the trivial one is using a rolling window (FIFO) that discard tokens older than n_ctx. AFAIK, the contexts in the KV memory are not position (row) dependent, and we should be able to overwrite old contexts by wrapping-around the index.
ChatGPT told me that another common strategy uses a LRU priority queue. Querying a context is not a binary but analog process. I guess that some sort of threshold can be used to call when a specific context is queried. Or maybe, derive some sort of score using an exponential smoothing of the past Queries \dot Key similarities.
I browsed some of the references provided by ChatGPT on this matter and they seemed to be mostly hallucinated; So I wouldn't trust it. It would be very helpful if someone knowledgeable could offer their perspective.
Hmm, I think yes - we need to shift the KV cache. I haven't implemented this yet in any of the
ggmlexamples. And when the context is full - stop increasingn_past.
Hi @ggerganov! (and anyone else interested :eyes:) I wanted to reach out to you because I've been working on implementing the idea you mentioned about shifting the KV cache to support infinite output mode in LLaMA. I've run some experiments, and it seems like we might need to rethink the approach.
I implementing shifting for the KV cache (and I triple checked everything), but after the window was shifted by a single token, the model started to output garbage. After a lot of testing and frustration, it hit me: positional encoding.
I realized that the embeddings stored in the memory_k and memory_v tensors indirectly store positionally encoded data. Therefore, shifting their contents messes with the semantics of those embeddings. While the code in llama_eval computes the values for cur_k and cur_v from the inpL before RoPE is applied, this is only true for the first layer. For every other layer, the contents of inpL are replaced by the output of the previous layer, meaning the tensors for every other layer already contain positional information in an indirect way.
I'm not entirely sure if my reasoning is correct, but my results seem to validate the idea that shifting the KV cache might not be enough. I'm reaching out to you (and anyone else who understands this) to see if you agree with my conclusions and to ask if you have any suggestions on how to proceed.
I think it would be really beneficial to have a way to slide the context window for LLaMA inference. This could be the key to unlocking a true ChatGPT clone with LLaMA. While having a fixed 2048 token window is a good start, being able to slide that window would enable the self-attention mechanism to remember much more than just the last 2048 tokens.
So anyway, sorry for the somewhat incoherent wall of text. My point is, I wanted to reach out because I'm out of ideas here, but I'm eager to hear your thoughts and any suggestions you (or others reading this) might have. :smile:
@setzer22 I only understand what you're saying in part, but reading your post made me think of something. In abstract terms, would it be possible in some way, instead of having a context that shifts, to have two contexts and do some sort of swap strategy?
I'm sure that carries its own issues. Due to my ignorance I can't be more specific, but I'm throwing the idea out there just in case it applies in some way or inspires you to think of something else.
@setzer22 That's true - it's not as simple as we initially thought. How does it work in the Python code? Maybe we get an idea from there
to have two contexts and do some sort of swap strategy?
@tjohnman I'm unsure what you mean by a swap strategy here. My first guess is that swapping things out wouldn't work. It sounds similar to clearing the full context window, unless I'm miunsderstanding something. :sweat_smile:
That's true - it's not as simple as we initially thought. How does it work in the Python code? Maybe we get an idea from there
@ggerganov I've only done a very quick look to the python code (assuming you mean https://github.com/facebookresearch/llama/), but I haven't seen anything referring to a sliding window, so I'm not sure if that's implemented there.
@setzer22 Yes, it's a very naïve idea I was proposing in hopes that more knowledgeable minds would be inspired somehow by it. If (or until) the context window can be shifted properly somehow, maybe a good compromise could be to use a new, cleared context but carrying over information from the previous one.
Again, perhaps it's a very naïve solution, but what about:
- Save the last n tokens.
- Clear the context window.
- Prompt/prime it using the saved tokens.
Would something like this work? Or perhaps something similar to this idea: https://twitter.com/miolini/status/1635559164297752577
Leveraging the model itself to summarize previous information and seeding a brand new context with it every time it fills up.
@ggerganov I've only done a very quick look to the python code (assuming you mean https://github.com/facebookresearch/llama/), but I haven't seen anything referring to a sliding window, so I'm not sure if that's implemented there.
So far, I haven't seen it used in fb/llama. I've been searching for a reference implementation of RoPE-based inference with a sliding window, but I haven't had any luck finding one. It would be great to have one to learn from.
I'm still going over the RoPE paper paper and haven't quite figured out how it relates to the ggml implementation. But I think we'll need to do some shuffling for the i2 < ne2 dimension.
(Edit: with this "Position Information in Transformers" review, it's easier to see how all the pieces fit together.)
How about looking at what https://github.com/oobabooga/text-generation-webui does? It can also do 4bit llama with GPUs & CPUs and has an infinite chatbot mode too. It's way more work to setup than llama.cpp, and llama.cpp w/ llama-30b seems to perform better with an M1 pro than a 3090 can with oobabooga, so I'm looking forward to llama.cpp getting this feature. @oobabooga is also pretty friendly.
it seems there are two possible solutions
swap idea: while using the model take half of the input tokens and start building inference over them in the background. When out of context size swap two the second instance. The model will loose all memory of before the new context window. This -: this is computationally intensive +: I am confident that this is technically possible
rolling context the problem here is positional encoding. the original positional encoding for transformers is not abel to do rolling context because the transformer is trained on fixed encodings. Something like the Alibi encoding might work for this. We cant choose the encoding because facebook chose Rope as mentioned during training. Rotational positional encoding sounds promising but after I skimmed over the paper it doesn't seem probable to me that RopE would support somehing like that. +: would be highly elegant. +: as mentioned the model could carry over information from past the context window. -: It seems unlikely to me that its possible (im not confident in that prediction)
@DKormann Forgive me because I'm still a total layman when it comes to the terminology. When you say "building inference" you mean using those tokens for prediction? The idea I got in very simple terms:
- You have two contexts.
- When you get to 50% capacity on the first one, start filling up the second one in parallel.
- When you get to 100% capacity on the first, switch to the second (which is now at 50%).
- Create a new one.
- Repeat.
This effectively means your context is half of the size, but you can keep going forever like this. Would this work?
@tjohnman first of im just a layman too:) 'building inference' is just me trying to describe that you need to feed the previous tokens into the model again. I understand the approach you are suggesting. I want to stress that feeding 50% of the context size into the model again requires half of the computation again. As soon as you swap to the next instance this instance is allready half full right? so you will need to start filling up a new instance again. Every token will go through the model twice (apart from the tokens in the very beggining). So it will double the computational effort. I not sure how hard it would be to have a model operate on two contexts simultaneously.
It would be nice to fill the second context while wating for user input. Maybe one could hide a lot of the extra cost from the user
Doubling costs is no joke. Unless this can be optimized somehow.
Building inference on the second context while waiting for input is not a bad idea, but if I compare the time I spend reading the output versus typing input, I don’t think that will be enough.
I can live with having the speed halved and RAM usage doubled when using the 7B model because it generates quite fast. But I don’t know if the effort of implementing it is worth it if the performance is going to be prohibitive in most use cases.
Perhaps someone with knowledge about the internals of the process can shed some light on whether this could be implemented in a saner way.
Please correct me if I'm wrong since I'm yet to fully grasp LLM's in general and I'm still very much in the learning phase.
If I understood correctly, basically the general idea of these models is to infer the next token by answering this question: "Given the previous token stream X, what should be the next token?" using probability analysis.
And that includes the end-of-stream token, correct? So when the calculation ends up predicting "stop here", it stops there.
Currently there is the option to ignore that token for the whole session since this commit https://github.com/ggerganov/llama.cpp/commit/50fae10d0339f2bd639f69dd679c0201d939a265
What is your opinion on including "magic keywords" for the interactive mode to control the variables. That could be extended to other variables too, but what I'm specifically thinking about here is having a "continue" magic keyword. So in the normal operation mode (not overridden by --ignore-eos), after reaching the end-of-stream token and control being given back to the user, inputting "continue" wouldn't put that in the input stream but rather skip the eos token and continue from there, once. It'd be more flexible than having a global variable for the whole session.
While I can't be obviously certain, it seems to me that this is how openai's interactive chatgpt demo does it.
This would only solve the "reached end-of-stream token too early" problem though while the other half is the need to have a sliding context window to be able to have a infinite output.
Again, please feel free to correct me if I understood something wrong.
Infinite generation should now be supported. The current implementation works like this:
- Keep generating until the context
n_ctx(i.e. 2048) becomes full - When full, set
n_past == n_keepwheren_keepis a user-provided parameter. By default, it is 0. Can be set to something in order to get a "static" prompt. See examples/chat.sh how we make Bob instructions to be a "static" prompt. You can observe the "static" prompt by adding the--verbose-promptargument - These
n_keeptokens are instantly available thanks to the KV cache, so no need to recompute anything so far - Next, we also pick half of the
n_ctx - n_keeptokens that were generated last and insert them for re-inference. This is currently happening serially so there is a delay when it occurs
For example, this command should generate text forever:
make -j && ./main -m ./models/7B/ggml-model-q4_0.bin -p "I believe the meaning of life is" -c 512 -t 8 -n -1 --ignore-eos
Infinite generation should now be supported. The current implementation works like this:
- Keep generating until the context
n_ctx(i.e. 2048) becomes full- When full, set
n_past == n_keepwhere `n_keep is a user-provided parameter. By default, it is 0. Can be set to something in order to get a "static" prompt. See examples/chat.sh how we make Bob instructions to be a "static" prompt- These
n_keeptokens are instantly available thanks to the KV cache, so no need to recompute anything so far- Next, we also pick half of the
n_ctx - n_keeptokens that were generated last and insert them for re-inference. This is currently happening serially so there is a delay when it occursFor example, this command should generate text forever:
make -j && ./main -m ./models/7B/ggml-model-q4_0.bin -p "I believe the meaning of life is" -c 512 -t 8 -n -1 --ignore-eos
This is great. Gonna test it out soon. One question, if --keep isn't specified or 0, does it just use the initial prompt's size or is the initial prompt discarded once n_ctx is reached? (I think the former is the case unless I'm misinterpreting https://github.com/ggerganov/llama.cpp/commit/e2d490dafd860eaaaf9aa8008ab790527d556daf#diff-2d3599a9fad195f2c3c60bd06691bc1815325b3560b5feda41a91fa71194e805R201?)
If --keep 0 there is no "static" prompt. So when the context is full with n_ctx tokens, we will pick the second half of them [n_ctx/2, n_ctx] and use that as a new prompt. The initial prompt that has been provided will eventually disappear after one or more swaps / rotations of the context.
Currently, the new context is constructed as n_keep + last (n_ctx - n_keep)/2 tokens, but this can also become a user-provided parameter. For example, instead of always picking half of the tokens, we can pick a specific number of tokens or a percentage. It's easy to extend in such ways.