👨‍💻 Awesome Code Benchmark

A comprehensive code domain benchmark review of LLM researches.

News

• 🔥🔥 [2025-09-22] Featured Benchmarks:

  🔥LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering from Salesforce AI Research

  🔥CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects from Ant Group

• 🔥🔥 [2025-08-29] Featured Benchmarks:

  🔥A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code from Tencent

  🔥GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging from UCAS

• 🔥🔥 [2025-08-22] Featured Benchmarks:

  🔥TRACY: Benchmarking Execution Efficiency of LLM-Based Code Translation from Peking University

  🔥BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models from University of Science and Technology of China

• 🔥🔥 [2025-08-16] Featured Benchmarks:

  🔥AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators from Hunyuan Team, Tencent

  🔥Dynamic Benchmark Construction for Evaluating Large Language Models on Real-World Codes from Beihang University

  🔥STEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution Reasoning from ByteDance

• 🔥🔥 [2025-07-23] Featured Benchmarks:

  🔥SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories? from Xi’an Jiaotong University and TikTok

  🔥CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks from ASUS Intelligent Cloud Services

  🔥Multilingual Multimodal Software Developer for Code Generation from Beihang University

  🔥CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance from Amazon Web Service

  🔥SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks from SberAI

  🔥IFEvalCode: Controlled Code Generation from Beihang University

  🔥Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security from Government Technology Agency

  🔥MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts? from University of Illinois Urbana-Champaign

  🔥Turning the Tide: Repository-based Code Reflection from Beihang University

• 🔥🔥 [2025-07-13] Featured Benchmarks:

  🔥CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks from Purdue University

  🔥ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation from Tencent Hunyuan Team

  🔥CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark from Shanghai Jiao Tong University

  🔥Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs from Provable Responsible AI and Data Analytics (PRADA) Lab

  🔥Model Editing for LLMs4Code: How Far are We? from National University of Defense Technology

  🔥VeriBench: Benchmarking Large Language Models for Verilog Code Generation and Design Synthesis from Indian Institute Of Technology Gandhinagar

  🔥ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness from Imperial College London United Kingdom

  🔥Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation from Chinese Academy of Sciences

• [2025-04-18] We add Github Stars for each banchmark.
• [2025-04-13] We add Code Security & Robustness benchmarks. 
• [2025-04-06] We add Code Hallucinations benchmarks. 
• [2025-03-29] We have crawled all the articles related to code benchmarks in the past five years. 
• [2025-03-17] We add Code Version (Version-specific code generation) benchmarks. 
• [2025-03-16] A thorough review of code domain benchmarks for LLM research has been released. 

Table of Content

• Code Completion & Code Generation
• Code Efficiency
• CodeFix & Bug-Fix
• Code Reasoning & Understanding
• Code Hallucination
• Data science
• Text2SQL
• MultiModal Code Tasks
• Code Security & Robustness
• Code Translation
• Code Version
• Multi & Other Dimension
• Industry Code Generation

Survey

1. Software Development Life Cycle Perspective A Survey of Benchmarks for Code Large Language Models and Agents from Xi’an Jiaotong University

2. Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks from Zhejiang University

3. A Survey on Large Language Model Benchmarks from Shenzhen Key Laboratory for High Performance Data Mining

🚀 Top Code Benchmark

Code Completion & Code Generation

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
HumanEval	Evaluating Large Language Models Trained on Code	Arxiv 2021/07	Github	🤗Dataset
MBPP	Program Synthesis with Large Language Models	Arxiv 2021/08	Github	🤗Dataset
DyCodeEval	DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination	ICML 2025	Github	🤗Dataset
PPM	PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models	FSE 2024	Github	🤗Dataset
APPS	Measuring Coding Challenge Competence With APPS	NeurIPS 2021	Github	🤗Dataset
CodeContests	Competition-Level Code Generation with AlphaCode	Science 2022	Github	Dataset
MultiPL-E	MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation	TSE 2023	Github	🤗Dataset
MCoNaLa	MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages	EACL 2023 Findings	Github	🤗Dataset
LCC	LongCoder: A Long-Range Pre-trained Language Model for Code Completion	ICML 2023	Github	Dataset
CodeClarQA	Python Code Generation by Asking Clarification Questions	ACL 2023	Github	Dataset
EvalPlus	Is Your Code Generated by Chat{GPT} Really Correct? Rigorous Evaluation of Large Language Models for Code Generation	NeurIPS 2023	Github	🤗Dataset 📊LeaderBoard
CrossCodeEval	CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion	NeurIPS 2023	Github	Dataset
ODEX	Execution-Based Evaluation for Open-Domain Code Generation	EMNLP 2023 Findings	Github	Dataset
RepoBench	RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems	ICLR 2024	Github	🤗Dataset
CatCoder	Enhancing Repository-Level Code Generation with Integrated Contextual Information	Arxiv 2024/06
StudentEval	StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code	ACL 2024 Findings	Github	🤗Dataset
DevEval	DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories	ACL 2024	Github	🤗Dataset
CoderEval	CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models	ICSE 2024	Github
ConCodeEval	ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages	Arxiv 2024/07
CodeScope	CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation	ACL 2024	Github	📊LeaderBoard 🤗Dataset
OOP	OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models	ACL 2024 Findings	Github	🤗Dataset
L2CEval	L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models	TACL 2024
HumanExtension	Exploring Language Model's Code Generation Ability with Auxiliary Functions	NAACL 2024 Findings	Github	🤗Dataset
LLM4Decompile	LLM4Decompile: Decompiling Binary Code with Large Language Models	EMNLP 2024	Github	🤗Dataset
PYCOMMITS	Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing	ICLR 2024	Github	Dataset
CodeAgentBench	CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges	ACL 2024
SAFIM	Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks	ICML 2024	Github	🤗Dataset
BigCodeBench	BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions	ICLR 2025	Github	🤗Dataset 📊LeaderBoard
EvoCodeBench	EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories	NeurIPS 2025	Github	🤗Dataset
DynaCode	DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation	Arxiv 2025/03
	A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs	EASE 2025
LeetCodeDataset	LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs	Arxiv 2025/04	Github	🤗Dataset
CodeFlowBench	CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation	Arxiv 2025/04	Github	🤗Dataset
CodeMixBench	CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts	Arxiv 2025/05		🤗Dataset
CPRet	CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming	Arxiv 2025/05	Github
ELABORATION	ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming	ACL 2025	Github
OSS-Bench	OSS-Bench: Benchmark Generator for Coding LLMs	Arxiv 2025/05	Github	🤗Dataset 📊LeaderBoard
VERINA	VERINA: Benchmarking Verifiable Code Generation	Arxiv 2025/05	Github	🤗Dataset
OIBench	OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics	Arxiv 2025/06		🤗Dataset
IFEvalCode	IFEvalCode: Controlled Code Generation	Arxiv 2025/07	Github	🌐Website
CodeEval Pro	HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation	ACL 2025	Github	🤗Dataset 🌐Website 📊LeaderBoard
Code2Bench	Dynamic Benchmark Construction for Evaluating Large Language Models on Real-World Codes	Arxiv 2025/08	Github	🌐Website
STEPWISE-CODEX-Bench	STEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution Reasoning	Arxiv 2025/08
AutoCodeBench	AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators	Arxiv 2025/08	Github	🤗Dataset 🌐Website 📊LeaderBoard

Code Efficiency

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
EvalPerf	Evaluating Language Models for Efficient Code Generation	COLM 2024	Github	🤗Dataset 🌐Website
EffiBench	EffiBench: Benchmarking the Efficiency of Automatically Generated Code	NeurIPS 2024	Github
Mercury	Mercury: A Code Efficiency Benchmark for Code Large Language Models	NeurIPS 2024	Github	🤗Dataset
ECCO	ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?	EMNLP 2024	Github	🤗Dataset
PIE	Learning Performance-Improving Code Edits	ICLR 2024	Github	🌐Website
ENAMEL	How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark	ICLR 2025	Github	🤗Dataset
	Improving Assembly Code Performance with Large Language Models via Reinforcement Learning	Arxiv 2025/05
EFFIBENCH-X	EFFIBENCH-X:A Multi-Language Benchmark fo rMeasuring Effciency ofLLM.Generated Code	Arxiv 2025/05	Github	🤗Dataset
PERFFORGE	Synthesizing Performance Constraints for Evaluating and Improving Code Efficiency	Arxiv 2025/05
SWE-Perf	SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?	Arxiv 2025/07	Github	🤗Dataset 🌐Website
TRACY	TRACY: Benchmarking Execution Efficiency of LLM-Based Code Translation	Arxiv 2025/08

CodeFix & Bug-Fix

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
Buggy-HumanEval&Buggy-FixEval	Large Language Models of Code Fail at Completing Code with Potential Bugs	NeurIPS 2023	Github	Dataset
SWT-Bench	SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents	NeurIPS 2024	Github	🌐Website
HumanEvalPack	OctoPack: Instruction Tuning Code Large Language Models	ICLR 2024	Github	🤗Dataset
SWE-bench	SWE-bench: Can Language Models Resolve Real-World GitHub Issues?	ICLR 2024	Github	🌐Website
GitBug-Java	GitBug-Java: A Reproducible Benchmark of Recent Java Bugs	MSR 2024	Github	🤗Dataset 🌐Website
GitBug-Actions	GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions	ICSE 2024 Demo	Github	▶️Video
RepoBugs	When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?	ICSE 2024 Industry Track
RepoFixEval	RepoFixEval: A Repository-Level Program Repair Benchmark From Issue Discovering to Bug Fixing	OpenReview 2024	Link
DebugBench	DebugBench: Evaluating Debugging Capability of Large Language Models	ACL 2024	Github	🤗Dataset
Multi-Bug	Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging	EMNLP 2024 Findings	Github
Coffee-Gym	Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code	EMNLP 2024		🤗Dataset
INTERVENOR	INTERVENOR: Prompt the Coding Ability of Large Language Models with the Interactive Chain of Repairing	ACL 2024 Findings	Github
StatType-SO	ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMs	TOSEM 2024
LiveCodeBench	LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code	ICLR 2025	Github	🤗Dataset 🌐Website 📊LeaderBoard
COAST	COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis	NAACL 2025	Github	🤗Dataset
SWE-bench Multimodal	SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?	ICLR 2025	Github	🤗Dataset 🌐Website
FeedbackEval	FeedbackEval A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks		Github
CVE-Bench	CVE-Bench:Benchmarking LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities	NAACL 2025	Github	Dataset
OmniGIRL	OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution	ISSTA 2025	Github	🤗Dataset 📊LeaderBoard
LongSWE-Bench	LongCodeBench: Evaluating Coding LLMs at 1M Context Windows	Arxiv 2025/05		🤗Dataset
VADER	VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation	Arxiv 2025/06	Github
Breakpoint	Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents	Arxiv 2025/05
MLDebugging	MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios	Arxiv 2025/06	Github
Skywork-SWE	Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs	Arxiv 2025/06
SWE-MERA	SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks	Arxiv 2025/07	Github	🤗Dataset 🌐Website
CodeFuse-CR-Bench	CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects	Arxiv 2025/09

Code Reasoning & Understanding

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
GenCodeSearchNet	GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding	EMNLP 2023	Github	🤗Dataset
CRUXEval	CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution	Arxiv 2024/01	Github	📊LeaderBoard
Poor-CodeSumEval	How Effectively Do Code Language Models Understand Poor-Readability Code?	ASE 2024	Github	🤗Dataset
CodeScope	CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation	ACL 2024	Github	📊LeaderBoard 🤗Dataset
CodeJudge-Eval	CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?	COLING 2025	Github
CodeMMLU	CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities of CodeLLMs	ICLR 2025	Github	🤗Dataset 🌐Website 📊LeaderBoard
LongCodeQA	LongCodeBench: Evaluating Coding LLMs at 1M Context Windows	Arxiv 2025/05		🤗Dataset
CTF-Code	Success is in the Details: Evaluate and Enhance Details Sensitivity of Code	Arxiv 2025/05
CodeSense	CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning	Arxiv 2025/06	Github	🤗Dataset📊LeaderBoard
CETBench	CETBench: A Novel Dataset constructed via Transformations over	Arxiv 2025/06
ICPC-Eval	ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests	Arxiv 2025/06	Github	🤗Dataset
CoQuIR	CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval	Arxiv 2025/06	Github
OJBench	OJBench: A Competition Level Code Benchmark For Large Language Models	Arxiv 2025/06
CORE	CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks	Arxiv 2025/07
CLMEEval	Model Editing for LLMs4Code: How Far are We?	ICSE 2025	Github	🤗Dataset
LONGCODEU	LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding	ACL 2025
LiveRepoReflection	Turning the Tide: Repository-based Code Reflection	Arxiv 2025/07	Github	🌐Website 📊LeaderBoard
LoCoBench	LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering	Arxiv 2025/09	Github

Code Hallucination

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
HALLUCODE	Exploring and Evaluating Hallucinations in LLM-Powered Code Generation	Arxiv 2024/04
Collu-Bench	Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code	Arxiv 2024/10		🤗Dataset
CodeHalu	CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification	AAAI 2025	Github	🤗Dataset
APIHulBench	Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware	FSE 25	Github
THINK	THINK: Tackling API Hallucinations in LLMs via Injecting Knowledge	SANER 2025	Github	🤗Dataset

Data science

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
DS-1000	DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation	ICML 2023	Github	🤗Dataset 🌐HomePage
ARCADE	Natural Language to Code Generation in Interactive Data Science Notebooks	ACL 2023	Github	Dataset
DA-Code	DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models	EMNLP 2024	Github	🤗Dataset 🌐Website
MatPlotBench	MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization	ACL 2024 Findings	Github	🤗Dataset
DataSciBench	DataSciBench: An LLM Agent Benchmark for Data Science	ArXiv 2025/02	Github
DSBench	DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?	ICLR 2025	Github	🤗Dataset
DSCodeBench	DS-Bench: A Realistic Benchmark for Data Science Code Generation	Arxiv 2025/05	Github

Text2SQL

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
Spider	Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task	EMNLP 2018	Github	🌐Website
SParC	SParC: Cross-Domain Semantic Parsing in Context	ACL 2019	Github	🌐Website
CoSQL	CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases	EMNLP 2019	Github	🌐Website
Spider-DK	Exploring underexplored limitations of crossdomain text-to-sql generalization	EMNLP 2021	Github
Spider-Syn	Towards robustness of text-to-SQL models against synonym substitution	ACL 2021	Github
Spider-Realistic	Structure-Grounded Pretraining for Text-to-SQL	NAACL 2021		Dataset
BIRD	Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs	NeurIPS 2023	Github	🌐Website
Dr.Spider	Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness	ICLR 2023	Github
BookSQL	BookSQL: A Large Scale Text-to-SQL Dataset for Accounting Domain	NAACL 2024	Github	Dataset
Archer	Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoning	EACL 2024		🌐Website
SecureSQL	SecureSQL: Evaluating Data Leakage of Large Language Models as Natural Language Interfaces to Databases	EMNLP 2024 Findings	Github	Dataset
Spider 2.0	Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows	ICLR 2025	Github	🌐Website
SNAILS	SNAILS: Schema Naming Assessments for Improved LLM-Based SQL Inference	PACMMOD 2025	Github
SQL2Text	Semantic Captioning: Benchmark Dataset and Graph-Aware Few-Shot In-Context Learning for SQL2Text	COLING 2025	Github	Dataset

MultiModal Code Tasks

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
MMCode	MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems	EMNLP 2024	Github	🤗Dataset
Drawing Pandas	Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code	ArXiv 2024/12	Github	🤗Dataset
Web2Code	Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs	NeurIPS 2024	Github	🤗Dataset 🌐Website
VGBench	VGBench: Evaluating Large Language Models on Vector Graphics Understanding and Generation	EMNLP 2024	Github	🤗Dataset
SVGEditBench	SVGEditBench: A Benchmark Dataset for Quantitative Assessment of LLM's SVG Editing Capabilities	CVPR2024 workshop	Github	🤗Dataset
Plot2Code	Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots	Arxiv 2024/05	Github	🤗Dataset
HumanEval-V	HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks	ArXiv 2024/10	Github	🌐Website 📊LeaderBoard 🤗Dataset
WebSight-Test	WAFFLE: Multi-Modal Model for Automated Front-End Development	Arxiv 2024/10	Github	🤗Dataset
Sketch2Code	Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping	Arxiv 2024/10	Github	🌐Website
Interaction2Code	Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping	Arxiv 2024/11	Github	🤗Dataset 📊LeaderBoard
ScratchEval	ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges	Arxiv 2024/11	Github	🤗Dataset
MRWeb	MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs	Arxiv 2024/12	Github	🤗Dataset
Image2Struct	Image2Struct: Benchmarking Structure Extraction for Vision-Language Models	NeurIPS 2024	Github	🌐Website 🤗Dataset
BigDocs-Bench	BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks	ICLR 2025		🤗Dataset 🌐Website
WebCode2M	WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs	WWW 2025	Github	🌐Website 🤗Dataset
Design2Code	Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering	NAACL 2025	Github	🤗Dataset
DiagramGenBenchmark	From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing	CVPR 2025	Github	🌐Website 🤗Dataset
ChartMimic	ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation	ICLR 2025	Github	🌐Website 🤗Dataset
SVG-Bench	StarVector: Generating Scalable Vector Graphics Code from Images and Text	CVPR 2025	Github	🌐Website 🤗Dataset
LLM4SVG	Empowering LLMs to Understand and Generate Complex Vector Graphics	CVPR 2025	Github	🌐Website
ChartCoder	ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation	Arxiv 2025/01	Github	🤗Dataset
Code-Vision	Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities	Arxiv 2025/02
Flame-React-Eval	Advancing vision-language models in front-end development via data synthesis	Arxiv 2025/03	Github	🤗Dataset
vTikZ	LLM Code Customization with Visual Results: A Benchmark on TikZ	EASE 2025
Plot2XML	Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation	Arxiv 2025/04
Flow2Code	Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability	Arxiv 2025/06	Github
DesignBench	DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation	Arxiv 2025/06	Github	🤗Dataset
WebUIBench	WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal	Arxiv 2025/06	Github	🤗Dataset
FrontendBench	FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation	Arxiv 2025/06
ArtifactsBench	ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation	Arxiv 2025/07	Github	🌐Website 🤗Dataset📊 Leaderboard
MVB	Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation	ICCAD 2024	Github
M^2 EVAL	Multilingual Multimodal Software Developer for Code Generation	Arxiv 2025/07	Github	🤗Dataset

Code Security & Robustness

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
COCO	COCO: Testing Code Generation Systems via Concretized Instructions	Arxiv 2023/08	Github
ReCode	ReCode: Robustness Evaluation of Code Generation Models	ACL 2023	Github	Dataset
RedCode	RedCode: Risky Code Execution and Generation Benchmark for Code Agents	NeurIPS 2024	Github	🌐Website 📊LeaderBoard
CodeWMBench	CodeWMBench: An Automated Benchmark for Code Watermarking Evaluation	ACM-TURC 2024	Github
RMCBench	RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code	ASE 2024	Github	🤗Dataset
PyP4LLMSec	Benchmarking the Security Aspect of Large Language Model-Based Code Generation	ICSE 2024	Github	Dataset
CWE-Bench-Java	IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities	Arxiv 2024/05	Github
CyberSecEval 3	CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models	Arxiv 2024/08	Github	Dataset
CS-Eval	CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity	Arxiv 2024/11	Github	🤗Dataset
SecBench	SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity	Arxiv 2024/12		Dataset 🌐Website
aiXamine	aiXamine: Simplified LLM Safety and Security	Arxiv 2025/04		🌐Website
SafeGenBench	SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code	Arxiv 2025/06
CodeMirage	CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs	Arxiv 2025/06
SEC-bench	SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks	Arxiv 2025/06	Github	🤗Dataset📊LeaderBoard
RAS-Eval	RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments	Arxiv 2025/06	Github
JsDeObsBench	JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation	CCS 2025	Github	📊Leaderboard
CIRCLE	Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security	Arxiv 2025/07		🤗Dataset
MOCHA	MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?	Arxiv 2025/07	Github	🤗Dataset
A.S.E	A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code	Arxiv 2025/08

Code Translation

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
TransCoder	Unsupervised Translation of Programming Languages	NeurIPS 2020	Github(deprecated) Github(new)	Dataset
AVATAR	AVATAR: A Parallel Corpus for Java-Python Program Translation	ACL Findings 2023	Github	Dataset
G-TransEval	On the Evaluation of Neural Code Translation: Taxonomy and Benchmark	ASE 2023	Github	🤗Dataset
CodeTransOcean	CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation	EMNLP 2023	Github	🤗Dataset
xCodeEval	XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval	ACL 2024	Github	🤗Dataset
PolyHumanEval	Unraveling the Potential of Large Language Models in Code Translation: How Far Are We?	APSEC 2024	Github	🤗Dataset
RustRepoTrans	Repository-level Code Translation Benchmark Targeting Rust	Arxiv 2024/11	Github	🤗Dataset
ClassEval-T	Escalating LLM-based Code Translation Benchmarking into the Class-level Era	Arxiv 2024-11	Github	🤗Dataset
TRANSREPO-BENCH	Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluation	Arxiv 2025/01	Github	🤗Dataset
LongTrans	Enhancing LLMs in Long Code Translation through Instrumentation and Program State Alignment	Arxiv 2025/04
CRUST-Bench	CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation	Arxiv 2025/04	Github	Dataset

Code Version

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
CodeUpdateEval	Automatically Recommend Code Updates: Are We There Yet?	TOSEM 2024	Github	🤗Dataset
JavaVersionGenBench	On the Generalizability of Deep Learning-based Code Completion Across Programming Language Versions	ICPC 2024	Github	🤗Dataset
VersiCode	VersiCode: Towards Version-controllable Code Generation	Arxiv 2024/10	Github	🌐Website 🤗Dataset
GitChameleon	GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models	Arxiv 2024/11	Github	🤗Dataset
LLM-Deprecated-APl	LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion	ICSE 2025	Github	🤗Dataset
LibEvolutionEval	LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation	NAACL 2025		🌐Website
CodeUpdateArena	CodeUpdateArena: Benchmarking Knowledge Editing on API Updates	Arxiv 2025/02	Github	🤗Dataset
RustEvo2	RustEvo2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation	Arxiv 2025/03	Github	🤗Dataset
CODEMENV	CODEMENV: Benchmarking Large Language Models on Code Migration	ACL 2025 Findings	Github	🤗Dataset

Multi & Other Dimension

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
Stack-Repo	RepoFusion: Training Code Models to Understand Your Repository	Arxiv 2023/06	Github	🤗Dataset
MultiNL-H	Improving Natural Language Capability of Code Large Language Model	Arxiv 2024/01	Github
HumanEvalPack	OctoPack: Instruction Tuning Code Large Language Models	ICLR 2024	Github	🤗Dataset
CodeBenchGen	CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks	Arxiv 2024/04	Github	Dataset
X-HumanEval-X	Exploring Multi-Lingual Bias of Large Code Models in Code Generation	Arxiv 2024/04
RACE	Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models	Arxiv 2024/07	Github	📊LeaderBoard
RealWorld-Bench	What's Wrong with Your Code Generated by Large Language Models? An Extensive Study	Arxiv 2024/07
APPS+	StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback	ACL 2024	Github	Dataset
InfiBench	InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models	NeurIPS 2024	Github	🌐Website
RobustAPI	Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation	AAAI 2024	Github	🤗Dataset
EvoEval	Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM	COLM 2024	Github
CodeScope	CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation	ACL 2024	Github	📊LeaderBoard 🤗Dataset
AssertionBench	AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation	NAACL 2025	Github
REval	Evaluating Large Language Models with Runtime Behavior of Program Execution	ICSE 2025	Github	📊LeaderBoard
LiveCodeBench	LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code	ICLR 2025	Github	🤗Dataset 🌐Website 📊LeaderBoard
SWE-PolyBench	SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents	Arxiv 2025/04	Github	🌐Website 🤗Dataset
Paper2Code	Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning	Arxiv 2025/04	Github	🤗Dataset
LiCoEval	LiCoEval: Evaluating LLMs on License Compliance in Code Generation	ICSE 2025	Github	Dataset
CoCo-Bench	CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation	Arxiv 2025/04
CodeRepetEval	Rethinking Repetition Problems of LLMs in Code Generation	ACL 2025	Github
WebGen-Bench	WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch	Arxiv 2025/03	Github	🤗Dataset
DecompileBench	DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios	Arxiv 2025/05	Github
CLEVER	CLEVER:A Curated Benchmark for Formally Verified	Arxiv 2025/05	Github	🤗Dataset
ResearchCodeBench	ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code	Arxiv 2025/06
CoreCodeBench	CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark	Arxiv 2025/07	Github	🤗Dataset
TeXpert	TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs	Arxiv 2025/06	Github
AUTOEXPERIMENT	From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking	Arxiv 2025/06	Github	🤗Dataset
CodeAssistBench	CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance	Arxiv 2025/07
CodeJudgeBench	CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks	Arxiv 2025/07		🤗Dataset
BinMetric	BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models	IJCAI 2025
GitTaskBench	GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging	Arxiv 2025/08	Github	🌐Website

Industry Code Generation

Benchmark	Paper	Date	Github	Dataset & Website & LeaderBoard
VerilogEval	VerilogEval Evaluating Large Language Models for Verilog Code Generation	ICCAD 2023	Github	🤗Dataset
VGen	Benchmarking Large Language Models for Automated Verilog RTL Code Generation	DATE 2023	Github	🤗Dataset
RTLLM	RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model	ASPDAC 2024	Github	🤗Dataset
LLM4PLC	LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systems	ICSE 2024	Github	🌐Website
Agents4PLC	Agents4PLC: Automating Closed-loop PLC Code Generation and Verification in Industrial Control Systems using LLM-based Agents	Arxiv 2024/10	Github	🤗Dataset
	A Multi-Agent Framework for Extensible Structured Text Generation in PLCs	Arxiv 2024/12
OpenLLM-RTL	OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generation	ICCAD 2024	Github	🤗Dataset
MG-Verilog	MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generation	ISLAD 2024	Github
RTL-Repo	RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projects	LAD 2024	Github	🤗Dataset
MetRex	MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs	ASPDAC 2025	Github	🤗Dataset
ResBench	ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness	Arxiv 2025/03	Github
ComplexVCoder	ComplexVCoder: An LLM-Driven Framework for Systematic Generation of Complex Verilog Code	Arxiv 2025/04
VeriBench	VeriBench: Benchmarking Large Language Models for Verilog Code Generation and Design Synthesis	ISCAS 2025