Whats the max amount of input text that can be supplied ?

[FiveTechSoft]

Anyone has made some tests using \ at the end of each line ? how large can it be ?

[FiveTechSoft]

I read 2048 tokens on discord. I will check it. I do appreciate if someone else can check it and share info about it

[pirate486743186]

There's a bug, it crashes if you exceed the limit.

[loss-and-quick]

I read 2048 tokens on discord. I will check it. I do appreciate if someone else can check it and share info about it

You are right, i checked limit via serge

See logs below:

RETURN CODE -11

main: warning: model does not support context sizes greater than 2048 tokens (4056 specified);expect poor results
main: seed = 1680468736
llama_model_load: loading model from '/usr/src/app/weights/gpt4all.bin' - please wait ...
llama_model_load: n_vocab = 32001
llama_model_load: n_ctx = 4056
llama_model_load: n_embd = 4096
llama_model_load: n_mult = 256
llama_model_load: n_head = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot = 128
llama_model_load: f16 = 2
llama_model_load: n_ff = 11008
llama_model_load: n_parts = 1
llama_model_load: type = 1
llama_model_load: ggml map size = 4017.70 MB
llama_model_load: ggml ctx size = 81.25 KB
llama_model_load: mem required = 5809.78 MB (+ 1026.00 MB per state)
llama_model_load: loading tensors from '/usr/src/app/weights/gpt4all.bin'
llama_model_load: model size = 4017.27 MB / num tensors = 291
llama_init_from_file: kv self size = 2028.00 MB

system_info: n_threads = 4 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
sampling: temp = 0.100000, top_k = 50, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000
generate: n_ctx = 4056, n_batch = 8, n_predict = 4056, n_keep = 0

[PocoLibre]

I tried this with 1110 tokens on colab, it will not take it. Nothing happens, the input ends immediately and nothing is printed. But if I do something like "Tell me a knock-knock joke" it works as expected.

[niansa]

Stale, please open a new issue if this is still relevant.