SWE-bench
SWE-bench is a benchmark that tests whether AI systems can resolve real GitHub issues from open-source Python repositories. A model is given an issue and the codebase, and its patch is scored by whether the project's own tests pass afterwards.
It became the headline coding benchmark because it is closer to real work than puzzle-style tests: multi-file changes in unfamiliar code, judged by an objective signal the model cannot argue with.
It still has known limits. It is Python and open-source, so it says little about your TypeScript monorepo or your internal frameworks. Scores are sensitive to the scaffolding around the model, so a number reflects a whole system rather than a model alone. And popular repositories risk contamination from training data.
What this means in practice
Useful for tracking whether the field is improving, close to useless for choosing between the tools on this site — they all reach similar frontier models, and what differs is context handling and workflow, which no benchmark measures.
Prompts for the tools this applies to
Related terms
- LLM evals
LLM evals are systematic tests of a model or AI feature's output quality: a set of inputs, a definition of a good response, and a scoring method.
- AI code generation
AI code generation is producing working code from a natural-language description — a function, a component, a module, or an entire application.