speculative-decoding topic

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

intel

[NeurIPS'23] Speculative Decoding with Big Little Decoder

kssteven418

Codes for our paper "Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation" (EMNLP 2023 Findings)

hemingkx

Large-scale LLM inference engine

PygmalionAI

scalable and robust tree-based speculative decoding algorithm

Infini-AI-Lab

[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

Infini-AI-Lab

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)

SafeAILab

minimal C implementation of speculative decoding based on llama2.c

mscheong01

REST: Retrieval-Based Speculative Decoding, NAACL 2024

FasterDecoding

Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan et al. 2023.

romsto