Longer and infinite output

[leszekhanusz]

If we use -n 1000000 to have a very long output (for a story for example),
it stops generating quite fast, after around 30 lines, probably because of this line of code.

It would be nice if we could have longer outputs and also the possibility to have infinite output, stopping only on Ctrl-C.
We could maybe specify that -n 0 will trigger that infinite output mode.
That issue is a bit related to issue #23

[eous]

Depending on how much memory you have you can increase the context size to get longer outputs. On a 64gb machine I was able to have a 12k context with the 7B model and 2k context with the 65B model. You can change it here

[matthew-mcallister]

Huh... I thought the context size was determined when the model was trained due to the positional encoding used. (I am only a layman.) But #78 is still useful for when you eventually hit the limit of your context, right?

[drewcrawford]

When trying large contexts, I often encounter

aggml_new_tensor_impl: not enough space in the context's memory pool (needed 702840624, available 701883745)

I played around a bit with increasing ctx_size but that did not work, I suspect an underlying memory UB as the cause, as lldb seems to trap on some suspicious memory accesses

[Vent3st]

Depending on how much memory you have you can increase the context size to get longer outputs. On a 64gb machine I was able to have a 12k context with the 7B model and 2k context with the 65B model. You can change it here

Your link goes to this code snippet if (!llama_model_load(params.model, model, vocab, 512)) { // TODO: set context from user input ?? is this correct to change the context size?

[bitRAKE]

ggml_new_tensor_impl: not enough space in the context's memory pool (needed 704905456, available 704155676) Assertion failed: false, file ggml.c, line 2516

@drewcrawford for me, that error doesn't appear to be context size related. I've run the same prompt at different context sizes and they all fail.

[eous]

Typically if you get the not enough space in the context error you tried setting the context too large, though on the larger models I have had to tweak this line too. The math around memory allocation in this thing isn't perfectly scaling on the larger models and unfortunately my fork has substantially diverged from master and too lazy to work on merging.

llama_model_load: loading model from 'models/13B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 8192
llama_model_load: n_embd  = 5120
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 13824
llama_model_load: n_parts = 2
llama_model_load: ggml ctx size = 20562.56 MB
llama_model_load: memory_size = 12800.00 MB, n_mem = 327680

8192 context size on quantized 13B model

[ggerganov]

Yes, the math for computing necessary memory for the ggml buffers and graphs has to be updated.

[Piezoid]

What's the best way to enable infinite output? Can we just shift-out the old contexts in K and V tensors (along the n_ctx dim) when they are full, or is there a better approach?

[ggerganov]

Hmm, I think yes - we need to shift the KV cache. I haven't implemented this yet in any of the ggml examples.
And when the context is full - stop increasing n_past.

[bitRAKE]

The 65B model sometimes crashes with 2k, but usually works ...

llama_model_load: loading model from './models/65B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 2048
llama_model_load: n_embd  = 8192
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 64
llama_model_load: n_layer = 80
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 22016
llama_model_load: n_parts = 8
llama_model_load: ggml ctx size = 49157.73 MB
llama_model_load: memory_size = 10240.00 MB, n_mem = 163840

The increased context doesn't increase memory use (still under 41GB) - until the context gets filled.

Yet, on the 7B model the calculation overflow at the start for 16k or greater. Usually, 15360 will work.
llama_model_load: ggml ctx size = 17592186044337.35 MB

[eous]

eous@e0213e0 here is some easy stuff I pulled out of my local hackery if anyone wants to play with it