← ChoreCode leaderboard · 20260706T163525Z
error-message-chore
Make the invalid-flag error actionable
tiny_flags.flags.parse_flag raises ValueError('bad state') for invalid input, but its docstring promises a message that names the offending state and lists the valid states. Fix the error message to honor that contract for any invalid input. Return a unified diff only.
Forbidden cheap shots: modifying or deleting visible tests; hard-coding messages for specific invalid inputs instead of formatting the state
Results by model
8 run configs| Run config | Accepted | Accept rate | Hard fails |
|---|---|---|---|
| gemma4-26b | 8/8 | 100% | 0 |
| deepseek-v4-flash | 5/5 | 100% | 0 |
| glm-5.2 | 4/5 | 80% | 0 |
| kimi-k2.6 | 4/5 | 80% | 0 |
| minimax-m3 | 4/5 | 80% | 0 |
| qwen3.6-27b | 4/5 | 80% | 0 |
| deepseek-v4-pro | 3/5 | 60% | 0 |
| minimax-m2.7 | 3/5 | 60% | 0 |
Attempts
expand a row for the patch and logsdeepseek-v4-flash · rep 0 · accepted · 229→243 tok · $0.000064
no patch produced
deepseek-v4-flash · rep 0 · accepted · 229→190 tok · $0.000055
no patch produced
deepseek-v4-flash · rep 1 · accepted · 229→242 tok · $0.000064
no patch produced
deepseek-v4-flash · rep 1 · accepted · 229→223 tok · $0.000061
no patch produced
deepseek-v4-flash · rep 2 · accepted · 229→249 tok · $0.000065
no patch produced
deepseek-v4-pro · rep 0 · accepted · 229→251 tok · $0.00032
no patch produced
deepseek-v4-pro · rep 0 · accepted · 229→1004 tok · $0.0010
no patch produced
deepseek-v4-pro · rep 1 · patch_apply_failed · 229→400 tok · $0.00045
no patch produced
deepseek-v4-pro · rep 1 · accepted · 229→1132 tok · $0.0011
no patch produced
deepseek-v4-pro · rep 2 · patch_apply_failed · 229→271 tok · $0.00034
no patch produced
gemma4-26b · rep 0 · accepted · 254→6212 tok · $0.0021
no patch produced
gemma4-26b · rep 0 · accepted · 254→4575 tok · $0.0015
no patch produced
gemma4-26b · rep 0 · accepted · 254→4569 tok · $0.0015
no patch produced
gemma4-26b · rep 1 · accepted · 254→4279 tok · $0.0014
no patch produced
gemma4-26b · rep 1 · accepted · 254→5239 tok · $0.0017
no patch produced
gemma4-26b · rep 1 · accepted · 254→6325 tok · $0.0021
no patch produced
gemma4-26b · rep 2 · accepted · 254→2635 tok · $0.0009
no patch produced
gemma4-26b · rep 2 · accepted · 254→6201 tok · $0.0021
no patch produced
glm-5.2 · rep 0 · accepted · 230→355 tok · $0.0008
no patch produced
glm-5.2 · rep 0 · patch_apply_failed · 230→368 tok · $0.0008
no patch produced
glm-5.2 · rep 1 · accepted · 230→1479 tok · $0.0027
no patch produced
glm-5.2 · rep 1 · accepted · 230→335 tok · $0.0007
no patch produced
glm-5.2 · rep 2 · accepted · 230→286 tok · $0.0006
no patch produced
kimi-k2.6 · rep 0 · patch_apply_failed · 234→1796 tok · $0.0063
no patch produced
kimi-k2.6 · rep 0 · accepted · 234→2782 tok · $0.0096
no patch produced
kimi-k2.6 · rep 1 · accepted · 234→1388 tok · $0.0049
no patch produced
kimi-k2.6 · rep 1 · accepted · 234→2298 tok · $0.0080
no patch produced
kimi-k2.6 · rep 2 · accepted · 234→1734 tok · $0.0061
no patch produced
minimax-m2.7 · rep 0 · accepted · 235→200 tok · $0.00019
no patch produced
minimax-m2.7 · rep 0 · accepted · 235→390 tok · $0.00032
no patch produced
minimax-m2.7 · rep 1 · accepted · 235→202 tok · $0.00019
no patch produced
minimax-m2.7 · rep 1 · patch_apply_failed · 235→758 tok · $0.0006
no patch produced
minimax-m2.7 · rep 2 · patch_apply_failed · 235→343 tok · $0.00029
no patch produced
minimax-m3 · rep 0 · accepted · 391→257 tok · $0.00043
no patch produced
minimax-m3 · rep 0 · accepted · 391→200 tok · $0.00036
no patch produced
minimax-m3 · rep 1 · accepted · 391→212 tok · $0.00037
no patch produced
minimax-m3 · rep 1 · accepted · 391→186 tok · $0.00034
no patch produced
minimax-m3 · rep 2 · patch_apply_failed · 391→266 tok · $0.00044
no patch produced
qwen3.6-27b · rep 0 · accepted · 243→2742 tok · $0.0067
no patch produced
qwen3.6-27b · rep 0 · accepted · 243→2220 tok · $0.0054
no patch produced
qwen3.6-27b · rep 1 · accepted · 243→2818 tok · $0.0068
no patch produced
qwen3.6-27b · rep 1 · accepted · 243→2603 tok · $0.0063
no patch produced
qwen3.6-27b · rep 2 · patch_apply_failed · 243→2590 tok · $0.0063
no patch produced