[Feature request?]: Running larger models without quantization.

[acheong08]

Current error

[1]    11624 segmentation fault (core dumped)  ./llama -m ./models/13B/ggml-model-f16.bin -p  -t 8 --temp 0.5 --top_p 1 

[acheong08]

Everything works when quantized.

[gjmulder]

Current error

[1]    11624 segmentation fault (core dumped)  ./llama -m ./models/13B/ggml-model-f16.bin -p  -t 8 --temp 0.5 --top_p 1 

Check whether you have enough memory:

Size	Precision	Approx. RAM req.
13B	fp16	25GB
30B	fp16	62GB
65B	fp16	122GB

You can add swap on Linux, but even using NVME it will be very slow due to the random access patterns of the code.

[acheong08]

Even my primary memory + swap is not enough.

[gjmulder]

As per the comments from @MarkSchmidty in issue #53 use the 4bit quantized models to reduce your memory requirements by approx. 4X. The loss in quality should be negligible.

[acheong08]

Yes. The quantized models work perfectly for me. I just wanted to test out the other versions to check out how they perform

[gjmulder]

Slowly 😄

If there's a specific model size and prompt you need I can compare the 4bit to the f16. I'm currently exploring different option permutations in issue #69