Answer: The OpenAI API randomness doesn't seem to cause a large variance in the overall benchmark results, though exercises that are "on the bubble" are a bit random depending on which variant OpenAI returns. The benchmark attempts to be deterministic, but the OpenAI chat APIs are not deterministic, even at temperature=0. Status: Confirmed Entry: tool/aider/troubleshooting/benchmark-randomness Revision: 1 Digest: sha256:c2aab145509e145e (changes whenever the content changes) Timeline: Created: 2026-11-13T02:56:06Z Last changed: 2026-11-13T02:56:06Z Last verified: 2026-11-20T11:41:59Z Next check: 2026-11-21T11:41:59Z Check interval: Daily Expected change: No fixed schedule Data: Field | Value | Source | As of effect on overall results | The OpenAI API randomness doesn’t seem to cause a large variance in the overall benchmark results. | [1] | 2026-11-20 determinism at temperature | temperature=0 | [1] | 2026-11-20 variations per request | less than 5-10 variations | [1] | 2026-11-20 number of exercises | 133 | [1] | 2026-11-20 repeat runs | 10 times | [1] | 2026-11-20 model tested | gpt-3.5-turbo-0613 | [1] | 2026-11-20 Sources: [1] Aider. "aider.chat/docs/benchmarks.html". https://aider.chat/docs/benchmarks.html Role: primary. Accessed: 2026-11-20T11:41:59Z Quote: "The OpenAI API randomness doesn’t seem to cause a large variance in the overall benchmark results." (Supports: effect on overall results) Quote: "the OpenAI chat APIs are not deterministic, even at temperature=0" (Supports: determinism at temperature) Quote: "usually less than 5-10 variations" (Supports: variations per request) Quote: "it would be ideal to run all 133 exercises many times" (Supports: number of exercises) Quote: "I ran all 133 exercises 10 times each against gpt-3.5-turbo-0613" (Supports: repeat runs) Quote: "against gpt-3.5-turbo-0613 with the whole edit format" (Supports: model tested) Published: 2026-04-25T16:45:23Z Snapshot SHA-256: 7078f35e3da79f05d479f8070f08228ceb7e26dd3d343ea4aca6928e50b47b20 (captured 2026-11-13T02:56:06Z) Verification: Method: Automated extraction from [1] with deepseek-flash (JSON output, thinking disabled). Every value was checked by code to appear verbatim in a quoted passage of [1]. On this check the text of [1] was unchanged since 2026-11-13T02:56:06Z, so the values verified then were kept. Cross-check: not-available at 2026-11-20T11:41:59Z. No independent second source is configured for this entry Last successful check: 2026-11-20T11:41:59Z Last attempt: 2026-11-20T11:41:59Z Notes and limitations: Extracted automatically from a single official source (section "Randomness"). Related: https://latestn.com/tool/aider/troubleshooting/check-infinite-output-status How do I know if Aider is using infinite output? https://latestn.com/tool/aider/troubleshooting/docker-limitations What are the limitations of running Aider with Docker? https://latestn.com/tool/aider/troubleshooting/fix-aider-not-found-error How do I fix the "aider not found" error when running aider? https://latestn.com/tool/aider/settings/yaml-config-file How do I set Aider options in a .aider.conf.yml file? https://latestn.com/tool/aider/troubleshooting/aider-replit-libstdc-crash Why does Aider crash with a missing libstdc++.so.6 library on Replit? https://latestn.com/tool/aider/troubleshooting/avoid-dependency-conflicts Why do dependency versions matter for Aider and how can I avoid package conflicts? https://latestn.com/tool/ All entries in AI tools Change log: Time | Event | Change | Source 2026-11-13T02:56:06Z | added | Entry created | — Changes for this entry as a feed: https://latestn.com/tool/aider/troubleshooting/benchmark-randomness/latest Challenge this entry: Open challenges: None How: If a value here is wrong or out of date, send what is wrong and a link to evidence. Challenges are queued and checked against the evidence; they do not change the entry by themselves. If the entry changes as a result, the change log says so. Form: https://latestn.com/dispute/tool/aider/troubleshooting/benchmark-randomness POST: https://latestn.com/dispute/tool/aider/troubleshooting/benchmark-randomness with claim, evidence_url, and optionally field and contact (form-encoded or JSON) GET: https://latestn.com/dispute/tool/aider/troubleshooting/benchmark-randomness?claim={what is wrong}&evidence_url={link} How to cite: LatestN. (2026). How does randomness affect Aider's benchmark results? Revision 1, last verified 2026-11-20T11:41:59Z. https://latestn.com/tool/aider/troubleshooting/benchmark-randomness (accessed 2026-11-21T09:59:50Z). @misc{LatestN_tool_aider_troubleshooting_benchmark_randomness_r1, title = {How does randomness affect Aider's benchmark results?}, author = {{LatestN}}, year = {2026}, howpublished = {\url{https://latestn.com/tool/aider/troubleshooting/benchmark-randomness}}, note = {Revision 1; digest sha256:c2aab145509e145e; last verified 2026-11-20T11:41:59Z; accessed 2026-11-21T09:59:50Z} } Reuse: License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) Attribution: Copying and adapting are allowed, including commercially, if you credit LatestN and link to the entry you used. Each entry ends with a ready-made citation. URL: https://latestn.com/tool/aider/troubleshooting/benchmark-randomness