Answer: Aider's GPT code editing benchmark uses 133 practice exercises from the Exercism python repository, where GPT must read instructions, implement the provided function/class skeletons, and pass all unit tests; the benchmark measures what percentage of the 133 exercises are completed successfully. Status: Confirmed Entry: tool/aider/features/benchmark-methodology Revision: 1 Digest: sha256:b469e0cf79b5044a (changes whenever the content changes) Timeline: Created: 2026-11-13T02:56:06Z Last changed: 2026-11-13T02:56:06Z Last verified: 2026-11-20T11:41:58Z Next check: 2026-11-21T11:41:58Z Check interval: Daily Expected change: No fixed schedule Data: Field | Value | Unit | Source | As of number of exercises | 133 | exercises | [1] | 2026-11-20 success metric | what percentage of the 133 exercises are completed successfully | | [1] | 2026-11-20 test error lines sent | first 50 lines | lines | [1] | 2026-11-20 goal | read the instructions, implement the provided function/class skeletons and pass all the unit tests | | [1] | 2026-11-20 Sources: [1] Aider. "aider.chat/docs/benchmarks.html". https://aider.chat/docs/benchmarks.html Role: primary. Accessed: 2026-11-20T11:41:58Z Quote: "The benchmark uses 133 practice exercises from the Exercism python repository" (Supports: number of exercises) Quote: "The benchmark measures what percentage of the 133 exercises are completed successfully, causing all the associated unit tests to pass." (Supports: success metric) Quote: "It only sends the first 50 lines of test errors to try and avoid exceeding the context window of the smaller models." (Supports: test error lines sent) Quote: "The goal is for GPT to read the instructions, implement the provided function/class skeletons and pass all the unit tests." (Supports: goal) Published: 2026-04-25T16:45:23Z Snapshot SHA-256: 7078f35e3da79f05d479f8070f08228ceb7e26dd3d343ea4aca6928e50b47b20 (captured 2026-11-13T02:56:06Z) Verification: Method: Automated extraction from [1] with deepseek-flash (JSON output, thinking disabled). Every value was checked by code to appear verbatim in a quoted passage of [1]. On this check the text of [1] was unchanged since 2026-11-13T02:56:06Z, so the values verified then were kept. Cross-check: not-available at 2026-11-20T11:41:58Z. No independent second source is configured for this entry Last successful check: 2026-11-20T11:41:58Z Last attempt: 2026-11-20T11:41:58Z Notes and limitations: Extracted automatically from a single official source (section "The benchmark"). Related: https://latestn.com/tool/aider/features/benchmark-results-notes What should I know when interpreting Aider's benchmark leaderboard results? https://latestn.com/tool/aider/features/community-contributed-conventions Where can I find community contributed coding conventions for Aider? https://latestn.com/tool/aider/features/compiled-languages How does Aider handle linting and testing for compiled languages? https://latestn.com/tool/aider/features/add-web-pages-to-chat How do I add web pages to an Aider chat? https://latestn.com/tool/aider/features/architect-mode-editor-model What is Aider's architect mode and how does the editor model work? https://latestn.com/tool/aider/features/architect-mode-for-edits How do I enable architect mode in Aider to improve file editing? https://latestn.com/tool/ All entries in AI tools Change log: Time | Event | Change | Source 2026-11-13T02:56:06Z | added | Entry created | — Changes for this entry as a feed: https://latestn.com/tool/aider/features/benchmark-methodology/latest Challenge this entry: Open challenges: None How: If a value here is wrong or out of date, send what is wrong and a link to evidence. Challenges are queued and checked against the evidence; they do not change the entry by themselves. If the entry changes as a result, the change log says so. Form: https://latestn.com/dispute/tool/aider/features/benchmark-methodology POST: https://latestn.com/dispute/tool/aider/features/benchmark-methodology with claim, evidence_url, and optionally field and contact (form-encoded or JSON) GET: https://latestn.com/dispute/tool/aider/features/benchmark-methodology?claim={what is wrong}&evidence_url={link} How to cite: LatestN. (2026). How does Aider's GPT code editing benchmark work? Revision 1, last verified 2026-11-20T11:41:58Z. https://latestn.com/tool/aider/features/benchmark-methodology (accessed 2026-11-21T09:53:45Z). @misc{LatestN_tool_aider_features_benchmark_methodology_r1, title = {How does Aider's GPT code editing benchmark work?}, author = {{LatestN}}, year = {2026}, howpublished = {\url{https://latestn.com/tool/aider/features/benchmark-methodology}}, note = {Revision 1; digest sha256:b469e0cf79b5044a; last verified 2026-11-20T11:41:58Z; accessed 2026-11-21T09:53:45Z} } Reuse: License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) Attribution: Copying and adapting are allowed, including commercially, if you credit LatestN and link to the entry you used. Each entry ends with a ready-made citation. URL: https://latestn.com/tool/aider/features/benchmark-methodology