← ChoreCode leaderboard · 20260706T163525Z
error_messages
1 public chores · 35/43 accepted · 0 hard fails
Tasks
category coverage| Task | Title | Accepted | Hard fails |
|---|---|---|---|
| error-message-chore | Make the invalid-flag error actionable | 35/43 (81%) | 0 |
Results
model × task| Task | Run config | Accepted | Hard fails |
|---|---|---|---|
| error-message-chore | deepseek-v4-flash | 5/5 | 0 |
| error-message-chore | deepseek-v4-pro | 3/5 | 0 |
| error-message-chore | gemma4-26b | 8/8 | 0 |
| error-message-chore | glm-5.2 | 4/5 | 0 |
| error-message-chore | kimi-k2.6 | 4/5 | 0 |
| error-message-chore | minimax-m2.7 | 3/5 | 0 |
| error-message-chore | minimax-m3 | 4/5 | 0 |
| error-message-chore | qwen3.6-27b | 4/5 | 0 |
Attempts
evidence for this categoryerror-message-chore · deepseek-v4-flash · rep 0 · accepted · 229→243 tok · $0.000064
no patch produced
error-message-chore · deepseek-v4-flash · rep 0 · accepted · 229→190 tok · $0.000055
no patch produced
error-message-chore · deepseek-v4-flash · rep 1 · accepted · 229→242 tok · $0.000064
no patch produced
error-message-chore · deepseek-v4-flash · rep 1 · accepted · 229→223 tok · $0.000061
no patch produced
error-message-chore · deepseek-v4-flash · rep 2 · accepted · 229→249 tok · $0.000065
no patch produced
error-message-chore · deepseek-v4-pro · rep 0 · accepted · 229→251 tok · $0.00032
no patch produced
error-message-chore · deepseek-v4-pro · rep 0 · accepted · 229→1004 tok · $0.0010
no patch produced
error-message-chore · deepseek-v4-pro · rep 1 · patch_apply_failed · 229→400 tok · $0.00045
no patch produced
error-message-chore · deepseek-v4-pro · rep 1 · accepted · 229→1132 tok · $0.0011
no patch produced
error-message-chore · deepseek-v4-pro · rep 2 · patch_apply_failed · 229→271 tok · $0.00034
no patch produced
error-message-chore · gemma4-26b · rep 0 · accepted · 254→6212 tok · $0.0021
no patch produced
error-message-chore · gemma4-26b · rep 0 · accepted · 254→4575 tok · $0.0015
no patch produced
error-message-chore · gemma4-26b · rep 0 · accepted · 254→4569 tok · $0.0015
no patch produced
error-message-chore · gemma4-26b · rep 1 · accepted · 254→4279 tok · $0.0014
no patch produced
error-message-chore · gemma4-26b · rep 1 · accepted · 254→5239 tok · $0.0017
no patch produced
error-message-chore · gemma4-26b · rep 1 · accepted · 254→6325 tok · $0.0021
no patch produced
error-message-chore · gemma4-26b · rep 2 · accepted · 254→2635 tok · $0.0009
no patch produced
error-message-chore · gemma4-26b · rep 2 · accepted · 254→6201 tok · $0.0021
no patch produced
error-message-chore · glm-5.2 · rep 0 · accepted · 230→355 tok · $0.0008
no patch produced
error-message-chore · glm-5.2 · rep 0 · patch_apply_failed · 230→368 tok · $0.0008
no patch produced
error-message-chore · glm-5.2 · rep 1 · accepted · 230→1479 tok · $0.0027
no patch produced
error-message-chore · glm-5.2 · rep 1 · accepted · 230→335 tok · $0.0007
no patch produced
error-message-chore · glm-5.2 · rep 2 · accepted · 230→286 tok · $0.0006
no patch produced
error-message-chore · kimi-k2.6 · rep 0 · patch_apply_failed · 234→1796 tok · $0.0063
no patch produced
error-message-chore · kimi-k2.6 · rep 0 · accepted · 234→2782 tok · $0.0096
no patch produced
error-message-chore · kimi-k2.6 · rep 1 · accepted · 234→1388 tok · $0.0049
no patch produced
error-message-chore · kimi-k2.6 · rep 1 · accepted · 234→2298 tok · $0.0080
no patch produced
error-message-chore · kimi-k2.6 · rep 2 · accepted · 234→1734 tok · $0.0061
no patch produced
error-message-chore · minimax-m2.7 · rep 0 · accepted · 235→200 tok · $0.00019
no patch produced
error-message-chore · minimax-m2.7 · rep 0 · accepted · 235→390 tok · $0.00032
no patch produced
error-message-chore · minimax-m2.7 · rep 1 · accepted · 235→202 tok · $0.00019
no patch produced
error-message-chore · minimax-m2.7 · rep 1 · patch_apply_failed · 235→758 tok · $0.0006
no patch produced
error-message-chore · minimax-m2.7 · rep 2 · patch_apply_failed · 235→343 tok · $0.00029
no patch produced
error-message-chore · minimax-m3 · rep 0 · accepted · 391→257 tok · $0.00043
no patch produced
error-message-chore · minimax-m3 · rep 0 · accepted · 391→200 tok · $0.00036
no patch produced
error-message-chore · minimax-m3 · rep 1 · accepted · 391→212 tok · $0.00037
no patch produced
error-message-chore · minimax-m3 · rep 1 · accepted · 391→186 tok · $0.00034
no patch produced
error-message-chore · minimax-m3 · rep 2 · patch_apply_failed · 391→266 tok · $0.00044
no patch produced
error-message-chore · qwen3.6-27b · rep 0 · accepted · 243→2742 tok · $0.0067
no patch produced
error-message-chore · qwen3.6-27b · rep 0 · accepted · 243→2220 tok · $0.0054
no patch produced
error-message-chore · qwen3.6-27b · rep 1 · accepted · 243→2818 tok · $0.0068
no patch produced
error-message-chore · qwen3.6-27b · rep 1 · accepted · 243→2603 tok · $0.0063
no patch produced
error-message-chore · qwen3.6-27b · rep 2 · patch_apply_failed · 243→2590 tok · $0.0063
no patch produced