Awesome_LLM_System-PaperList icon indicating copy to clipboard operation
Awesome_LLM_System-PaperList copied to clipboard

Published 20 hours ago • verified publisher icon galeselee
→

Metadata

Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inferenc...

description: >- Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on LLMs inference and serving.

Awesome_LLM_System-PaperList

Survey

Paper Keywords Institute (first) Publication Others
Full Stack Optimization for Transformer Inference: a Survey Hardware and software co-design UCB Arxiv
A survey of techniques for optimizing transformer inference Transformer optimization Iowa State Univeristy Journal of Systems Architecture
A Survey on Model Compression for Large Language Models Model Compression UCSD Arxiv
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems Optimization technique: quant, pruning, continuous batching, virtual memory CMU Arxiv
LLM Inference Unveiled: Survey and Roofline Model Insights Performance analysis Infinigence-AI Arxiv LLMViewer
LLM Inference Serving: Survey of Recent Advances and Opportunities Northeastern University Arxiv
Efficient Large Language Models: A Survey The Ohio State University Transactions on Machine Learning Research

Framework

Paper/OpenSource Project Keywords Institute (first) Publication Others
DeepSpeed Infernce: Enabling Efficient Inference of Transformer Models at Unprecedented Scale Deepspeed; Kerenl Fusion MicroSoft SC 2022 Github repo
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference Deepspeed; Split fuse MicroSoft Arxiv Github repo
Efficient Memory Management for Large Language Model Serving with PagedAttention vLLM; pagedAttention UCB SOSP 2023 Github repo
TensorRT-LLM/FastTransformer NVIDIA
lightLLM Shanghai Artifcial Intelligence Laboratory
MLC LLM TVM; Multi-platforms MLC-Team
Text-Generation-Inference(TGI) Huggingface
NanoFlow: Towards Optimal Large Language Model Serving Throughput Distributed, Parallel, and Cluster Computing University of Washington Arxiv Github repo
rtp-llm Alibaba Github repo
Efficiently Programming Large Language Models using SGLang Agent Language UCB Arxiv Github repo
HybridFlow: A Flexible and Efficient RLHF Framework RLHF Training ByteDance Eurosys 2024 Github Repo
ReaLHF: Optimized RLHF Training for Large Language Models through Parameter Reallocation RLHF Training THU Arxiv Github Repo
Enabling Parallelism Hot Switching for Efficient Training of Large Language Models Peking University SOSP 2024

Serving

Paper Keywords Institute (first) Publication Others
Fast Distributed Inference Serving for Large Language Models Distributed inference serving PKU Arxiv
AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving Pipeline Parallel; Auto parallel UCB OSDI 2023 Github repo
Orca: A Distributed Serving System for Transformer-Based Generative Models Continuous batching Seoul National University OSDI2022
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Multiple Decoding Heads Princeton University Arxiv Github repo
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU Consumer-grade GPU SJTU Arxiv Github repo
LLM in a flash: Efficient Large Language Model Inference with Limited Memory flash; Pruning Apple Arxiv
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline Length Perception NUS NeurIPS 2023 Github repo
S3: Increasing GPU Utilization during Generative Inference for Higher Throughput Harvard University Arxiv
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving Decouple PKU OSDI 2024
Splitwise: Efficient generative LLM inference using phase splitting Decouple UW ISCA 2024 Track issue
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Single GPU Stanford University Arxiv Github repo
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve Decouple GaTech OSDI 2024
SpotServe: Serving Generative Large Language Models on Preemptible Instances Preemptible GPU CMU ASPLOS 2024 Empty Github repo
SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification Tree-based Speculative CMU ASPLOS 2024
AttentionStore: Cost-effective Attention Reuse across Multi-turn Conversations in Large Language Model Serving Cache the multi-turn prefill KV-cache in host-DRAM and SSD NUS ATC 2024
MuxServe: Flexible Multiplexing for Efficient Multiple LLM Serving Use spatial-temporal multiplexing method to serve multi-LLMs MMLab Arxiv
PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference KV Cache Compression Shanghai Jiao Tong University Arxiv
You Only Cache Once: Decoder-Decoder Architectures for Language Models KV Cache Microsoft Research Arxiv
Better & Faster Large Language Models via Multi-token Prediction Multi-token Prediction Meta Arxiv
ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference Decouple Hanyang University ASPLOS 2024
Parrot: Efficient Serving of LLM-based Applications with Semantic Variable LLM Applications SJTU OSDI 2024
Fairness in Serving Large Language Models Fairness; LLM Serving UC Berkeley,Stanford University OSDI 2024
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving KV Cache Moonshot AI Github
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention Pre-fillingfor Long-Context
Dynamic Sparse Attention		 Microsoft Arxiv Github repo
MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool Memory Pool Huawei Arxiv
InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management sparisity Seoul National University OSDI 2024
Llumnix: Dynamic Scheduling for Large Language Model Serving Preemptible GPU Alibaba Group OSDI 2024
PUZZLE: Efficiently Aligning Large Language Models through Light-Weight Context Switch Multi-Agent Tsinghua University ATC 2024
SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention Sparsity; Long context PKU Arxiv
Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference Sparsity; Related token MIT ICML 2024
Accelerating Production LLMs with Combined Token/Embedding Speculators Speculative decoding IBM Research Arxiv Github repo
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference KV Cache Apple Arxiv
Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPU Attention Saddles,KV cache Shanghai Jiao Tong University Arxiv
TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text KV Cache for RAG Moore Threads AI Arxiv Github repo
Efficient Streaming Language Models with Attention Sinks StreamingLLM, Static sparsity MIT ICLR 2024 Github repo
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models sparsity attention UT Austin nips 2024
SparQ Attention: Bandwidth-Efficient LLM Inference sparsity attention GraphCore ICML 2024
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention sparsity attention msra arxiv
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval Vector Retrieval msra arxiv
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion KV Cache cross University of Chicago Eurosys
Epic: Efficient Position-Independent Context Caching for Serving Large Language Models Position independent PKU arxiv
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving KV Cache compression University of Chicage sigcomm
SCOPE:OptimizingKey-Value Cache Compression in Long-context Generation Separate handling of prefill and decoding KV Cache SEU arxiv 2024
FASTDECODE: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines Heterogeneous pipelines THU arxiv 2024

Operating System

Paper Keywords Institute(first) Publication Others
AIOS: LLM Agent Operating System OS; LLM Agent Rutgers University Arxiv

Transformer accelerate

Paper Keywords Institute (first) Publication Others
TurboTransformers: An Efficient GPU serving System For Transformer Models Tencent PPoPP 2021 Github repo
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness FlashAttention; Online Softmax Stanford University NeurIPS 2023 Github repo
FlashAttention2: Faster Attention with Better Parallelism and Work Partitioning Stanford University Arxiv Github repo
FlashDecoding++: Faster Large Language Model Inference on GPUs Softmax with Unified Maximum Value Tsinghua University Mlsys 2024
FlashFFTConv: Efficient Convolutions for Long Sentences with Tensor Cores FFT; TensorCore; Long Sentences Stanford University Arxiv Github repo
FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks Georgia Institute of Technology ASPLOS 2023
ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs Variable-Length Inputs UCR PPoPP 2022 Github repo
Fast Transformer Decoding: One Write-Head is All You Need MQA Google Arxiv
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints GQA Google Research ACL 2023
LightSeq: A High Performance Inference Library for Transformers ByteDance NAACL 2021 Github repo
LightSeq2: LightSeq2: Accelerated Training for Transformer-based Models on GPUs ByteDance SC 2022
Blockwise Parallel Transformer for Large Context Models Blockwise transformer UCB NeurIPS 2023 Github repo
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention Dynamic Memory Management Microsoft Research India Arxiv

Model Compression

Quant and Pruning

Paper Keywords Institute (first) Publication Others
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving SJTU mlsys 2024 Github repo
Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference Dynamic Compression NVIDIA Arxiv
Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric Algorithm-System Co-Design on Modern GPUs FP6 USYD ATC 2024
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration AWQ MIT mlsys 2024 bp
Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity Univeristy of Sydney VLDB 2024 Github repo
CLLMs: Consistency Large Language Models Consistency Shanghai Jiao Tong University Arxiv Github repo
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers ETH ICLR 2023
Optimal Brain Damage(OBD) Breaking work AT&T bell NIPS 2rd
Optimal Brain Surgeon:Extensions and performance comparisons Breaking work standford NIPS 1993
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression ETH NeurIPS 2020
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models MIT PMLR 2023
QuIP: 2-Bit Quantization of Large Language Models With Guarantees Cornell University NeurIPS 2023
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks Cornell University PMLR 2024
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models VQ MSRA EMNLP 2024
GPTVQ: The Blessing of Dimensionality for LLM Quantization VQ Qualcomm AI Research ICML 2024
PQCache: Product Quantization-based KVCache for Long Context LLM Inference PQ PKU arxiv 2024
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval ANNs MSRA arxiv 2024
Transformer-VQ: Linear-Time Transformers via Vector Quantization VQ Independent Researcher ICLR 2024
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache KVCache Rice University ICML 2024
QServe:W4A8KV4QuantizationandSystemCo-designforEfficientLLMServing Algorithm and system codesign MIT Arxiv2024
QTIP: Quantization with Trellises and Incoherence Processing VQ Cornell University 2024 nips spotlight
PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression Improve STE Yandex, HSE 2024 nips oral

Communication

Paper Keywords Institute (first) Publication Others
Overlap communication with dependent compuation via Decompostion in Large Deep Learning Models Overlap Google ASPLOS 2023
Efficiently scaling Transformer inference Scaling Google Mlsys 2023
Centauri: Enabling efficient scheduling for communication-computation overlap in large model training via communication communication partition PKU ASPLOS 2024

Energy

Paper Keywords Institute (first) Publication Others
Zeus: Understanding and Optimizing GPU energy Consumption of DNN Training Yale University NSDI 2023 Github repo
Power-aware Deep Learning Model Serving with μ-Serve UIUC ATC 2024
Characterizing Power Management Opportunities for LLMs in the Cloud LLM Microsoft Azure ASPLOS 2024
DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency LLM Serving Cluster UIUC Arxiv

Decentralized

Paper Keywords Institute (first) Publication Others
FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs Consumer-grade GPU HKBU Arxiv
Petals: Collaborative Inference and Fine-tuning of Large Models Yandex Arxiv

Serveless

Paper Keywords Institute (first) Publication Others
ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models cold boot The University of Edinburgh OSDI 2024 Empty Github
StreamBox: A Lightweight GPU SandBox for Serverless Inference Workflow HUST ATC 2024 Github

Trace

Paper Keywords Institute (first) Publication Others
Characterization of Large Language Model Development in the Datacenter Cluster trace(for LLM) ShangHai AI Lab NSDI 2024 Github
BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems GPT users trace HKUSTGZ Arxiv 2024 Github
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving Disaggregated trace Moonshot AI Github
Splitwise: Efficient generative LLM inference using phase splitting Disaggregated trace UW and microsoft ISCA 2024 Github Trace

← Metadata

156
Stars
6
Forks
Watchers

Owner

verified publisher icon galeselee

Metadata

Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inferenc...

Back