Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

SWE-bench series just aren't that great by today's standard, even Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark, I suspect the recent increase in the score for the Pro version probably also had similar behaviors.

But anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.

 help



> Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark

yeah that was the point of introducing the Pro version




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: