SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

TL;DR

SWE-bench turns real GitHub issues and repository states into tasks for evaluating whether language models can produce working software patches.

Why it matters

Repository-level tasks have become a key environment for coding agents and for training against execution-based, verifiable feedback.

Key findings

1. 01

   Real repositories require context navigation, code changes, and test-based verification.

2. 02

   Execution feedback creates a stronger signal than text-only judging for many coding tasks.

Scaling dimensions

Models, methods & benchmarks

Benchmarks

SWE-bench

Topics