← ChoreCode leaderboard · 20260706T163525Z
generated_artifact
2 public chores · 30/48 accepted · 1 hard fails
Tasks
category coverage| Task | Title | Accepted | Hard fails |
|---|---|---|---|
| generated-cli-help-chore | Regenerate CLI help after adding lint | 17/24 (71%) | 1 |
| generated-client-field-chore | Rename the generated API client field | 13/24 (54%) | 0 |
Results
model × task| Task | Run config | Accepted | Hard fails |
|---|---|---|---|
| generated-cli-help-chore | deepseek-v4-flash | 2/3 | 0 |
| generated-cli-help-chore | deepseek-v4-pro | 1/3 | 1 |
| generated-cli-help-chore | gemma4-26b | 3/3 | 0 |
| generated-cli-help-chore | glm-5.2 | 2/3 | 0 |
| generated-cli-help-chore | kimi-k2.6 | 3/3 | 0 |
| generated-cli-help-chore | minimax-m2.7 | 2/3 | 0 |
| generated-cli-help-chore | minimax-m3 | 1/3 | 0 |
| generated-cli-help-chore | qwen3.6-27b | 3/3 | 0 |
| generated-client-field-chore | deepseek-v4-flash | 1/3 | 0 |
| generated-client-field-chore | deepseek-v4-pro | 3/3 | 0 |
| generated-client-field-chore | gemma4-26b | 3/3 | 0 |
| generated-client-field-chore | glm-5.2 | 2/3 | 0 |
| generated-client-field-chore | kimi-k2.6 | 0/3 | 0 |
| generated-client-field-chore | minimax-m2.7 | 0/3 | 0 |
| generated-client-field-chore | minimax-m3 | 3/3 | 0 |
| generated-client-field-chore | qwen3.6-27b | 1/3 | 0 |
Attempts
evidence for this categorygenerated-cli-help-chore · deepseek-v4-flash · rep 0 · accepted · 189→670 tok · $0.00014
no patch produced
generated-cli-help-chore · deepseek-v4-flash · rep 1 · accepted · 189→347 tok · $0.000079
no patch produced
generated-cli-help-chore · deepseek-v4-flash · rep 2 · patch_apply_failed · 189→471 tok · $0.00010
no patch produced
generated-client-field-chore · deepseek-v4-flash · rep 0 · accepted · 229→420 tok · $0.000096
no patch produced
generated-client-field-chore · deepseek-v4-flash · rep 1 · patch_apply_failed · 229→493 tok · $0.00011
no patch produced
generated-client-field-chore · deepseek-v4-flash · rep 2 · patch_apply_failed · 229→646 tok · $0.00014
no patch produced
generated-cli-help-chore · deepseek-v4-pro · rep 0 · accepted · 189→493 tok · $0.0005
no patch produced
generated-cli-help-chore · deepseek-v4-pro · rep 1 · generated_drift, hidden_tests_failed · 189→374 tok · $0.00041
no patch produced
generated-cli-help-chore · deepseek-v4-pro · rep 2 · patch_apply_failed · 189→404 tok · $0.00043
no patch produced
generated-client-field-chore · deepseek-v4-pro · rep 0 · accepted · 229→636 tok · $0.0007
no patch produced
generated-client-field-chore · deepseek-v4-pro · rep 1 · accepted · 229→477 tok · $0.0005
no patch produced
generated-client-field-chore · deepseek-v4-pro · rep 2 · accepted · 229→608 tok · $0.0006
no patch produced
generated-cli-help-chore · gemma4-26b · rep 0 · accepted · 223→6303 tok · $0.0021
no patch produced
generated-cli-help-chore · gemma4-26b · rep 1 · accepted · 223→4142 tok · $0.0014
no patch produced
generated-cli-help-chore · gemma4-26b · rep 2 · accepted · 223→4368 tok · $0.0015
no patch produced
generated-client-field-chore · gemma4-26b · rep 0 · accepted · 275→4676 tok · $0.0016
no patch produced
generated-client-field-chore · gemma4-26b · rep 1 · accepted · 275→6074 tok · $0.0020
no patch produced
generated-client-field-chore · gemma4-26b · rep 2 · accepted · 275→1089 tok · $0.00038
no patch produced
generated-cli-help-chore · glm-5.2 · rep 0 · patch_apply_failed · 191→192 tok · $0.00044
no patch produced
generated-cli-help-chore · glm-5.2 · rep 1 · accepted · 191→206 tok · $0.00047
no patch produced
generated-cli-help-chore · glm-5.2 · rep 2 · accepted · 191→265 tok · $0.0006
no patch produced
generated-client-field-chore · glm-5.2 · rep 0 · patch_apply_failed · 235→296 tok · $0.0007
no patch produced
generated-client-field-chore · glm-5.2 · rep 1 · accepted · 235→167 tok · $0.00043
no patch produced
generated-client-field-chore · glm-5.2 · rep 2 · accepted · 235→237 tok · $0.0005
no patch produced
generated-cli-help-chore · kimi-k2.6 · rep 0 · accepted · 189→3492 tok · $0.0120
no patch produced
generated-cli-help-chore · kimi-k2.6 · rep 1 · accepted · 189→3044 tok · $0.0105
no patch produced
generated-cli-help-chore · kimi-k2.6 · rep 2 · accepted · 189→2230 tok · $0.0077
no patch produced
generated-client-field-chore · kimi-k2.6 · rep 0 · patch_apply_failed · 231→2760 tok · $0.0096
no patch produced
generated-client-field-chore · kimi-k2.6 · rep 1 · patch_apply_failed · 231→3997 tok · $0.0138
no patch produced
generated-client-field-chore · kimi-k2.6 · rep 2 · patch_apply_failed · 231→5615 tok · $0.0193
no patch produced
generated-cli-help-chore · minimax-m2.7 · rep 0 · patch_apply_failed · 0→334 tok · $0.00024
no patch produced
generated-cli-help-chore · minimax-m2.7 · rep 1 · accepted · 0→314 tok · $0.00023
no patch produced
generated-cli-help-chore · minimax-m2.7 · rep 2 · accepted · 191→340 tok · $0.00028
no patch produced
generated-client-field-chore · minimax-m2.7 · rep 0 · patch_apply_failed · 236→574 tok · $0.00046
no patch produced
generated-client-field-chore · minimax-m2.7 · rep 1 · patch_apply_failed · 236→678 tok · $0.0005
no patch produced
generated-client-field-chore · minimax-m2.7 · rep 2 · patch_apply_failed · 236→478 tok · $0.00039
no patch produced
generated-cli-help-chore · minimax-m3 · rep 0 · accepted · 347→277 tok · $0.00044
no patch produced
generated-cli-help-chore · minimax-m3 · rep 1 · patch_apply_failed · 347→1180 tok · $0.0015
no patch produced
generated-cli-help-chore · minimax-m3 · rep 2 · patch_apply_failed · 347→1019 tok · $0.0013
no patch produced
generated-client-field-chore · minimax-m3 · rep 0 · accepted · 392→809 tok · $0.0011
no patch produced
generated-client-field-chore · minimax-m3 · rep 1 · accepted · 392→492 tok · $0.0007
no patch produced
generated-client-field-chore · minimax-m3 · rep 2 · accepted · 392→383 tok · $0.0006
no patch produced
generated-cli-help-chore · qwen3.6-27b · rep 0 · accepted · 202→1919 tok · $0.0047
no patch produced
generated-cli-help-chore · qwen3.6-27b · rep 1 · accepted · 202→1592 tok · $0.0039
no patch produced
generated-cli-help-chore · qwen3.6-27b · rep 2 · accepted · 202→3222 tok · $0.0078
no patch produced
generated-client-field-chore · qwen3.6-27b · rep 0 · patch_apply_failed · 254→2472 tok · $0.0060
no patch produced
generated-client-field-chore · qwen3.6-27b · rep 1 · accepted · 254→2426 tok · $0.0059
no patch produced
generated-client-field-chore · qwen3.6-27b · rep 2 · patch_apply_failed · 254→2357 tok · $0.0057
no patch produced