SWE-bench

SWE-bench tests a real issue in an existing repository with tests and dependencies rather than generation of an isolated function. The Verified set is additionally filtered by people, but a score is still not equivalent to engineering work: tests may miss architecture, security and maintainability.