llama : lookahead decoding example

[wsxiaoys]

Claim providing 1.5~2x decoding speedup without a speculative model

Blog post: https://lmsys.org/blog/2023-11-21-lookahead-decoding/
Twitter thread: https://twitter.com/lmsysorg/status/1727056892671950887
Reference implementation: https://github.com/hao-ai-lab/LookaheadDecoding/tree/main

[bobqianic]

How does Medusa differ from this method?
https://github.com/FasterDecoding/Medusa

[someone13574]

How does Medusa differ from this method? https://github.com/FasterDecoding/Medusa

I believe (correct me if I'm wrong) that it doesn't require extra training of the model.

[KerfuffleV2]

How does Medusa differ from this method?

The blog link actually mentions Medusa specifically and then talks about how their approach is different.

[SlyEcho]

It certainly seems a little faster to me. From 30 t/s to 40 t/s on the LLaMA2-7B-chat example.

[shermansiu]

Also, lookahead decoding (LADE) seems to be constrained by the number of FLOPS available in consumer GPUs. I'm not sure how this will translate to CPU/RAM requirements, but whether LADE delivers improvements in performance seems to depend on how powerful your hardware is and whether the LADE parameters are optimized for your hardware.

huggingface/transformers#27649 (comment)

[ggerganov]

Example in #4207