👨‍💻 Awesome Code Benchmark

A comprehensive code domain benchmark review of LLM researches.

Taxonomy

This list organizes code benchmarks by primary capability and software-engineering workflow. Each benchmark entry includes compact metadata for task type, granularity, interaction pattern, and evaluation method.

Dimension	Values
Granularity	Function, API, Query / Database, File, Project, UI, Repository, Workflow
Interaction	Single-turn, Multi-turn, Agentic, Async, Multi-agent
Evaluation	Unit Tests, Execution, Performance, Human, LLM-as-Judge, Security Exploit, Economic
Environment	None, Sandbox, Terminal, Browser, IDE, CI
Freshness	Static, Dynamic, Held-out, Contamination-resistant

Surveys

Software Development Life Cycle Perspective A Survey of Benchmarks for Code Large Language Models and Agents from Xi’an Jiaotong University
Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks from Zhejiang University
A Survey on Large Language Model Benchmarks from Shenzhen Key Laboratory for High Performance Data Mining

🚀 Benchmark Categories

Repository & Agentic Software Engineering

ProjDevBench (2026): ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development - task: End-to-end project development - granularity: Project / Repository - interaction: Agentic - evaluation: Unit Tests / LLM-as-Judge - Github
HWE-Bench (2026): HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks - task: Hardware repository bug repair - granularity: Repository - interaction: Agentic - evaluation: Execution
SpecBench (2026): Measuring Reward Hacking in Long-Horizon Coding Agents - task: Reward hacking / spec compliance - granularity: Systems-level task (JSON parser → OS kernel) - interaction: Agentic - evaluation: Held-out vs visible tests - Github
IDE-Bench (2026): IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks - task: IDE-native software engineering - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution
RepoGenesis (2026): RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository - task: End-to-end microservice repository generation - granularity: Project / Repository - interaction: Agentic - evaluation: Unit Tests / Execution / Deployment - Github
SWE-Refactor (2026): SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring - task: Repository-level refactoring - granularity: Repository - interaction: Single-turn / Agentic - evaluation: Compilation / Unit Tests / Static Analysis
SWE-ContextBench (2026): SWE Context Bench: A Benchmark for Context Learning in Coding - task: Context reuse across related coding tasks - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Unit Tests / Execution / Cost
SWE-Bench Mobile (2026): SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications? - task: Industrial mobile application development - granularity: Repository / Workflow - interaction: Multimodal / Agentic - evaluation: Unit Tests / Execution - 🌐Website
RepoMod-Bench (2026): RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing - task: Repository modernization / migration - granularity: Project / Repository - interaction: Agentic - evaluation: Black-box Tests / Execution - Github
SWE-QA-Pro (2026): SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding - task: Agentic repository-level code understanding - granularity: Repository - interaction: Agentic - evaluation: QA Accuracy / Execution
SWE-STEPS (2026): Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution - task: Sequential software evolution - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Unit Tests / Execution / Static Analysis
RepoZero (2026): RepoZero: Can LLMs Generate a Code Repository from Scratch? - task: Repository generation from scratch - granularity: Project / Repository - interaction: Agentic - evaluation: Black-box Tests / Execution
VISTA (2026): VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents - task: Visual spec-to-web-app development - granularity: Project / Workflow - interaction: Multimodal / Agentic - evaluation: Browser Tests / Visual Similarity / DOM Matching - Github 🌐Website 🤗Dataset
RAMP (2026): Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems - task: Production runtime assessment - granularity: Project / Workflow - interaction: Agentic - evaluation: Execution / Runtime / Cost - 🌐Website
Claw-SWE-Bench (2026): Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks - task: Agent harness evaluation for issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution / Cost - Github 🤗Dataset
Dialogue SWE-Bench (2026): Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents - task: Dialogue-driven issue resolving - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Unit Tests / Dialogue Quality
SWE-Future (2026): SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents - task: Future-oriented repository task synthesis - granularity: Repository - interaction: Agentic - evaluation: Semantic Matching / Execution
SWE-Lancer (2025): SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? - task: Freelance software engineering - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution / Human / Economic - Github
Multi-SWE-bench (2025): Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving - task: Multilingual issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
SWE-Bench Pro (2025): SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? - task: Long-horizon issue resolving - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution
PRDBench (2025): Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation - task: PRD-driven project development - granularity: Project / Repository - interaction: Agentic - evaluation: Execution / LLM-as-Judge
LongCodeBench (2025): LongCodeBench: Evaluating Coding LLMs at 1M Context Windows - task: Long-context code comprehension and repair - granularity: Repository - interaction: Single-turn / Agentic - evaluation: Unit Tests / Execution / LLM-as-Judge - 🤗Dataset
SWE-Bench++ (2025): SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories - task: Repository-level bug fixing and feature requests - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
AInsteinBench (2025): AInsteinBench: Benchmarking Coding Agents on Scientific Repositories - task: Scientific repository development - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution
SWE-PolyBench (2025): SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents - task: Repository-level issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🌐Website 🤗Dataset
CoreCodeBench (2025): CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark - task: Repository-level software engineering - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution - Github 🤗Dataset
GitTaskBench (2025): GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging - task: Repository-level coding tasks - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution - Github 🌐Website
CodeAssistBench (2025): CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance - task: Multi-turn code assistance - granularity: Function / Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Human / LLM-as-Judge
SWE-Dev (2025): SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development - task: Feature-driven software development - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution - Github
SWE-Flow-Eval (2025): SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner - task: Test-driven incremental development - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution - Github
LoCoBench-Agent (2025): LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering - task: Long-context software engineering workflows - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Execution / Tool-use Metrics / LLM-as-Judge

Program Repair, Testing & Debugging

RepoExploreBench (2026): Planning to Explore: Curiosity-Driven Planning for LLM Test Generation - task: Iterative test generation - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
CI-Repair-Bench (2026): CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows - task: CI failure diagnosis and repair - granularity: Repository / Workflow - interaction: Agentic - evaluation: CI / Execution
ArkEval (2026): ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS - task: ArkTS automated program repair - granularity: Project / Repository - interaction: Single-turn / Agentic - evaluation: Unit Tests / Execution
HEJ-Robust (2026): HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair - task: Robust automated program repair - granularity: Function - interaction: Single-turn - evaluation: Unit Tests / Execution
FixtureEval (2026): Fixturize: Bridging the Fixture Gap in Test Generation - task: Test fixture classification and generation - granularity: Function / File - interaction: Agentic - evaluation: Unit Tests / Execution
RESTestBench (2026): RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements - task: Requirement-based REST API test generation - granularity: API / Workflow - interaction: Single-turn / Multi-turn - evaluation: Mutation Testing / Execution
QBugLM (2026): QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging - task: Quantum software debugging and repair - granularity: File / Project - interaction: Multi-agent - evaluation: Simulation / Execution
A11YBench (2026): A11YRepair: Bridging Web Accessibility Barriers via Knowledge-Enhanced Divide-and-Conquer Repair - task: Web accessibility repair - granularity: Project / Repository - interaction: Agentic - evaluation: Browser Tests / Execution
AgentDefect (2026): SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents - task: LLM-agent bug repair - granularity: File / Project - interaction: Agentic - evaluation: Unit Tests / Execution
HWE-Bench (2026): HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks - task: Hardware repository bug repair - granularity: Repository - interaction: Agentic - evaluation: Simulation / Execution
LiveCodeBench (2025): LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website 📊LeaderBoard
COAST (2025): COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset
SWE-bench Multimodal (2025): SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website
FeedbackEval (2025): FeedbackEval A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
CVE-Bench (2025): CVE-Bench:Benchmarking LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github Dataset
OmniGIRL (2025): OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 📊LeaderBoard
LongSWE-Bench (2025): LongCodeBench: Evaluating Coding LLMs at 1M Context Windows - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - 🤗Dataset
VADER (2025): VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
Breakpoint (2025): Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
MLDebugging (2025): MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
Skywork-SWE (2025): Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
SWE-MERA (2025): SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website
BuildBench (2025): BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software - task: Build and compilation repair - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution
SWT-Bench (2024): SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🌐Website
HumanEvalPack (2024): OctoPack: Instruction Tuning Code Large Language Models - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset
SWE-bench (2024): SWE-bench: Can Language Models Resolve Real-World GitHub Issues? - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🌐Website
GitBug-Java (2024): GitBug-Java: A Reproducible Benchmark of Recent Java Bugs - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website
GitBug-Actions (2024): GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github ▶️Video
RepoBugs (2024): When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done? - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
RepoFixEval (2024): RepoFixEval: A Repository-Level Program Repair Benchmark From Issue Discovering to Bug Fixing - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Link
DebugBench (2024): DebugBench: Evaluating Debugging Capability of Large Language Models - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset
Multi-Bug (2024): Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
Coffee-Gym (2024): Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - 🤗Dataset
INTERVENOR (2024): INTERVENOR: Prompt the Coding Ability of Large Language Models with the Interactive Chain of Repairing - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
StatType-SO (2024): ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMs - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
Buggy-HumanEval&Buggy-FixEval (2023): Large Language Models of Code Fail at Completing Code with Potential Bugs - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github Dataset

Security, Reliability & Robustness

RealSec-bench (2026): RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories - task: Secure code generation - granularity: Repository - interaction: Single-turn / Agentic - evaluation: SecurePass / SAST / Execution
HardSecBench (2026): HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation - task: Hardware and firmware secure code generation - granularity: Function / Project - interaction: Single-turn / Agentic - evaluation: Unit Tests / Security Exploit / Execution
TOSSS (2026): TOSSS: a CVE-based Software Security Benchmark for Large Language Models - task: Secure-vs-vulnerable code selection - granularity: Function / Snippet - interaction: Single-turn - evaluation: Security Score
Multi-LLMSecCodeEval (2026): Does Teaming-Up LLMs Improve Secure Code Generation? A Comprehensive Evaluation with Multi-LLMSecCodeEval - task: Secure code generation and remediation pipeline evaluation - granularity: Function / Workflow - interaction: Multi-agent - evaluation: SAST / Security Exploit / Execution
RealVuln (2026): RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code - task: Vulnerability detection in real-world code - granularity: Repository - interaction: Single-turn / Agentic - evaluation: Vulnerability Detection / F-score - 🌐Website
Delulu (2026): Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks - task: Code hallucination detection - granularity: Function / File - interaction: Single-turn - evaluation: Compilation / Execution / Human - Github
SEC-bench Pro (2026): SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks? - task: Long-horizon software security bug hunting - granularity: Repository / Workflow - interaction: Agentic - evaluation: PoC / Oracle / Execution
RewardHackingAgents (2026): RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents - task: Agent evaluation-integrity robustness - granularity: Project / Workflow - interaction: Agentic - evaluation: Integrity / Runtime / Execution
CodeHalu (2025): CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution - Github 🤗Dataset
APIHulBench (2025): Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution - Github
THINK (2025): THINK: Tackling API Hallucinations in LLMs via Injecting Knowledge - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution - Github 🤗Dataset
aiXamine (2025): aiXamine: Simplified LLM Safety and Security - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - 🌐Website
SafeGenBench (2025): SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution
CodeMirage (2025): CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution
SEC-bench (2025): SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🤗Dataset 📊LeaderBoard
RAS-Eval (2025): RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github
JsDeObsBench (2025): JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 📊Leaderboard
CIRCLE (2025): Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - 🤗Dataset
MOCHA (2025): MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts? - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🤗Dataset
A.S.E (2025): A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution
HALLUCODE (2024): Exploring and Evaluating Hallucinations in LLM-Powered Code Generation - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution
Collu-Bench (2024): Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution - 🤗Dataset
RedCode (2024): RedCode: Risky Code Execution and Generation Benchmark for Code Agents - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🌐Website 📊LeaderBoard
CodeWMBench (2024): CodeWMBench: An Automated Benchmark for Code Watermarking Evaluation - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github
RMCBench (2024): RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🤗Dataset
PyP4LLMSec (2024): Benchmarking the Security Aspect of Large Language Model-Based Code Generation - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github Dataset
CWE-Bench-Java (2024): IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github
CyberSecEval 3 (2024): CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github Dataset
CS-Eval (2024): CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🤗Dataset
SecBench (2024): SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Dataset 🌐Website
COCO (2023): COCO: Testing Code Generation Systems via Concretized Instructions - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github
ReCode (2023): ReCode: Robustness Evaluation of Code Generation Models - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github Dataset

Code Understanding, Search & Review

SWE-PRBench (2026): SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback - task: Pull request review - granularity: Repository / Workflow - interaction: Single-turn / Agentic - evaluation: LLM-as-Judge / Human
SWE-QA-Pro (2026): SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding - task: Repository-level code understanding - granularity: Repository - interaction: Agentic - evaluation: QA Accuracy / Execution
Code-QA-Bench (2026): Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA - task: Repository-level code QA - granularity: Repository - interaction: Agentic - evaluation: LLM-as-Judge / Human
CORE-Bench (2026): CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding - task: Repository-aware code retrieval - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Retrieval Metrics
CoREB (2026): Beyond Retrieval: A Multitask Benchmark and Model for Code Search - task: Code retrieval and reranking - granularity: Function / Repository - interaction: Single-turn - evaluation: nDCG / Retrieval Metrics
CLARC (2026): CLARC: C/C++ Benchmark for Robust Code Search - task: Robust code search - granularity: Function / Repository - interaction: Single-turn - evaluation: Retrieval Metrics - 🤗Dataset
R2Eval (2026): Evaluating LLMs Code Reasoning Under Real-World Context - task: Real-world code reasoning - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / Accuracy
CodeRQ-Bench (2026): Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks - task: Code reasoning quality evaluation - granularity: Function / File - interaction: Single-turn - evaluation: Human / LLM-as-Judge - Github
CodeGlance (2026): CodeGlance: Understanding Code Reasoning Challenges in LLMs through Multi-Dimensional Feature Analysis - task: Code behavior reasoning - granularity: Function / API - interaction: Single-turn - evaluation: Accuracy / Feature Analysis
code-logic-bench (2026): Imandra CodeLogician: Neuro-Symbolic Reasoning for Precise Analysis of Software Logic - task: Formal software logic reasoning - granularity: Function / Program - interaction: Agentic - evaluation: Formal Verification / Accuracy
CodeJudge-Eval (2025): CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding? - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github
CodeMMLU (2025): CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities of CodeLLMs - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset 🌐Website 📊LeaderBoard
LongCodeQA (2025): LongCodeBench: Evaluating Coding LLMs at 1M Context Windows - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - 🤗Dataset
CTF-Code (2025): Success is in the Details: Evaluate and Enhance Details Sensitivity of Code - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
CodeSense (2025): CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset 📊LeaderBoard
CETBench (2025): CETBench: A Novel Dataset constructed via Transformations over - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
ICPC-Eval (2025): ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
CoQuIR (2025): CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github
OJBench (2025): OJBench: A Competition Level Code Benchmark For Large Language Models - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
CORE (2025): CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
CLMEEval (2025): Model Editing for LLMs4Code: How Far are We? - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
LONGCODEU (2025): LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
LiveRepoReflection (2025): Turning the Tide: Repository-based Code Reflection - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🌐Website 📊LeaderBoard
LoCoBench (2025): LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github
CodeFuse-CR-Bench (2025): CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects - task: End-to-end code review - granularity: Repository / Workflow - interaction: Agentic - evaluation: LLM-as-Judge / Human
ContextCRBench (2025): Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice - task: Fine-grained code review - granularity: File / Repository - interaction: Single-turn / Agentic - evaluation: LLM-as-Judge / Human
CodeJudgeBench (2025): CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks - task: Code judging / evaluation - granularity: Function / Repository - interaction: Single-turn - evaluation: LLM-as-Judge / Human - 🤗Dataset
CRUXEval (2024): CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 📊LeaderBoard
Poor-CodeSumEval (2024): How Effectively Do Code Language Models Understand Poor-Readability Code? - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
CodeScope (2024): CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 📊LeaderBoard
🤗Dataset
GenCodeSearchNet (2023): GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset

Performance Optimization

CodegenBench (2026): CodegenBench: Can LLMs Write Efficient Code Across Architectures? - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Code Dataset
ENAMEL (2025): How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset
Improving Assembly Code Performance with Large Language Models via Reinforcement Learning (2025): Improving Assembly Code Performance with Large Language Models via Reinforcement Learning - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance
EFFIBENCH-X (2025): EFFIBENCH-X:A Multi-Language Benchmark fo rMeasuring Effciency ofLLM.Generated Code - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset
PERFFORGE (2025): Synthesizing Performance Constraints for Evaluating and Improving Code Efficiency - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance
SWE-Perf (2025): SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories? - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset 🌐Website
TRACY (2025): TRACY: Benchmarking Execution Efficiency of LLM-Based Code Translation - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance
EvalPerf (2024): Evaluating Language Models for Efficient Code Generation - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset 🌐Website
EffiBench (2024): EffiBench: Benchmarking the Efficiency of Automatically Generated Code - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github
Mercury (2024): Mercury: A Code Efficiency Benchmark for Code Large Language Models - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset
ECCO (2024): ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness? - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset
PIE (2024): Learning Performance-Improving Code Edits - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🌐Website

Frontend, UI & Visual-Interactive Development

BigDocs-Bench (2025): BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - 🤗Dataset
🌐Website
WebCode2M (2025): WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
🤗Dataset
Design2Code (2025): Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
DiagramGenBenchmark (2025): From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
🤗Dataset
ChartMimic (2025): ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website 🤗Dataset
SVG-Bench (2025): StarVector: Generating Scalable Vector Graphics Code from Images and Text - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
🤗Dataset
LLM4SVG (2025): Empowering LLMs to Understand and Generate Complex Vector Graphics - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
ChartCoder (2025): ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
Code-Vision (2025): Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge
Flame-React-Eval (2025): Advancing vision-language models in front-end development via data synthesis - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
vTikZ (2025): LLM Code Customization with Visual Results: A Benchmark on TikZ - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge
Plot2XML (2025): Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge
Flow2Code (2025): Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github
DesignBench (2025): DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
WebUIBench (2025): WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
FrontendBench (2025): FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge
ArtifactsBench (2025): ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website 🤗Dataset 📊 Leaderboard
M^2 EVAL (2025): Multilingual Multimodal Software Developer for Code Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
WebGen-Bench (2025): WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
MMCode (2024): MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
Drawing Pandas (2024): Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
Web2Code (2024): Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
🌐Website
VGBench (2024): VGBench: Evaluating Large Language Models on Vector Graphics Understanding and Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
SVGEditBench (2024): SVGEditBench: A Benchmark Dataset for Quantitative Assessment of LLM's SVG Editing Capabilities - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
Plot2Code (2024): Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
HumanEval-V (2024): HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
📊LeaderBoard
🤗Dataset
WebSight-Test (2024): WAFFLE: Multi-Modal Model for Automated Front-End Development - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
Sketch2Code (2024): Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
Interaction2Code (2024): Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
📊LeaderBoard
ScratchEval (2024): ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
MRWeb (2024): MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
Image2Struct (2024): Image2Struct: Benchmarking Structure Extraction for Vision-Language Models - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
🤗Dataset

Code Generation & Completion

DyCodeEval (2025): DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
BigCodeBench (2025): BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 📊LeaderBoard
EvoCodeBench (2025): EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
LibEvolutionEval (2025): LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation - task: Version-specific code completion - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution - Github 🌐Website
DynaCode (2025): DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs (2025): A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
LeetCodeDataset (2025): LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
CodeFlowBench (2025): CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
CodeMixBench (2025): CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - 🤗Dataset
CPRet (2025): CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github
ELABORATION (2025): ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github
OSS-Bench (2025): OSS-Bench: Benchmark Generator for Coding LLMs - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 📊LeaderBoard
VERINA (2025): VERINA: Benchmarking Verifiable Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
OIBench (2025): OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - 🤗Dataset
IFEvalCode (2025): IFEvalCode: Controlled Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🌐Website
CodeEval Pro (2025): HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website 📊LeaderBoard
Code2Bench (2025): Dynamic Benchmark Construction for Evaluating Large Language Models on Real-World Codes - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🌐Website
STEPWISE-CODEX-Bench (2025): STEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution Reasoning - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
AutoCodeBench (2025): AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website 📊LeaderBoard
SR-Eval (2025): SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
DataSciBench (2025): DataSciBench: An LLM Agent Benchmark for Data Science - task: Data science code generation - granularity: Project / Workflow - interaction: Single-turn / Agentic - evaluation: Execution - Github
Spider 2.0 (2025): Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows - task: Text-to-SQL - granularity: Query / Database - interaction: Single-turn - evaluation: Execution - Github 🌐Website
CRUST-Bench (2025): CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation - task: Code translation / migration - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github Dataset
Paper2Code (2025): Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning - task: Research code generation - granularity: Project / Repository - interaction: Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
PPM (2024): PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
RepoBench (2024): RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
CatCoder (2024): Enhancing Repository-Level Code Generation with Integrated Contextual Information - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
StudentEval (2024): StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
DevEval (2024): DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
CoderEval (2024): CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github
ConCodeEval (2024): ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
CodeScope (2024): CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 📊LeaderBoard
🤗Dataset
OOP (2024): OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
L2CEval (2024): L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
HumanExtension (2024): Exploring Language Model's Code Generation Ability with Auxiliary Functions - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
LLM4Decompile (2024): LLM4Decompile: Decompiling Binary Code with Large Language Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
PYCOMMITS (2024): Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
CodeAgentBench (2024): CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
SAFIM (2024): Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
RTLLM (2024): RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model - task: Domain-specific code generation - granularity: Project / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset
OpenLLM-RTL (2024): OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generation - task: Domain-specific code generation - granularity: Project / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset
MultiPL-E (2023): MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
MCoNaLa (2023): MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
LCC (2023): LongCoder: A Long-Range Pre-trained Language Model for Code Completion - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
CodeClarQA (2023): Python Code Generation by Asking Clarification Questions - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
EvalPlus (2023): Is Your Code Generated by Chat{GPT} Really Correct? Rigorous Evaluation of Large Language Models for Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 📊LeaderBoard
CrossCodeEval (2023): CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
ODEX (2023): Execution-Based Evaluation for Open-Domain Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
DS-1000 (2023): DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation - task: Data science code generation - granularity: Project / Workflow - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset 🌐HomePage
BIRD (2023): Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs - task: Text-to-SQL - granularity: Query / Database - interaction: Single-turn - evaluation: Execution - Github 🌐Website
CodeTransOcean (2023): CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation - task: Code translation / migration - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset
VerilogEval (2023): VerilogEval Evaluating Large Language Models for Verilog Code Generation - task: Domain-specific code generation - granularity: Project / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset
CodeContests (2022): Competition-Level Code Generation with AlphaCode - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
HumanEval (2021): Evaluating Large Language Models Trained on Code - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
MBPP (2021): Program Synthesis with Large Language Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
APPS (2021): Measuring Coding Challenge Competence With APPS - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
TransCoder (2020): Unsupervised Translation of Programming Languages - task: Code translation / migration - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github(deprecated) Github(new) Dataset