← ChoreCode leaderboard · 20260706T163525Z
gemma4-26b
gemma4:26b-mlx
* latency reflects local hardware and load
Stats
release-0.1.1| Model | gemma4:26b-mlx |
| Accepted | 84/108 (78%) |
| Hard fail rate | 2% |
| $ / accepted chore | $0.0017 |
| $ / attempt | $0.0013 |
| Tokens / accepted chore | 5411 |
| Tokens in / out (total) | 27.1k / 427.4k |
| Mean latency | 91.0 s * |
Cost is recorded input/output tokens multiplied by current OpenRouter pricing for the same model or weights.
Results by task
17 tasks| Task | Accepted | Hard fails | Failure reasons seen |
|---|---|---|---|
| bare-except-chore | 5/8 | 0 | patch_apply_failed |
| basic-function-off-by-one | 8/8 | 0 | — |
| config-noop-chore | 7/8 | 0 | patch_apply_failed |
| debug-cruft-chore | 8/8 | 0 | — |
| dependency-conflict-chore | 5/6 | 0 | patch_apply_failed |
| dependency-existing-retry-chore | 0/3 | 0 | existing_tests_failed, hidden_tests_failed, smoke_failed |
| dependency-stdlib-query-chore | 1/3 | 0 | patch_apply_failed |
| docstring-chore | 8/8 | 0 | — |
| error-message-chore | 8/8 | 0 | — |
| generated-cli-help-chore | 3/3 | 0 | — |
| generated-client-field-chore | 3/3 | 0 | — |
| nextflow-orchestration-chore | 0/6 | 2 | patch_apply_failed, forbidden_pattern, hidden_tests_failed, smoke_failed |
| project-manager-chore | 4/8 | 0 | patch_apply_failed |
| pyproject-edit-chore | 6/6 | 0 | — |
| tiny-python-type-chore | 4/8 | 0 | patch_apply_failed |
| type-annotation-chore | 8/8 | 0 | — |
| version-hallucination-chore | 6/6 | 0 | — |
Attempts
expand a row for the patch and logsbare-except-chore · rep 0 · accepted · 217→9885 tok · $0.0033
no patch produced
bare-except-chore · rep 0 · accepted · 217→11005 tok · $0.0036
no patch produced
bare-except-chore · rep 0 · accepted · 217→10019 tok · $0.0033
no patch produced
bare-except-chore · rep 1 · patch_apply_failed · 217→8419 tok · $0.0028
no patch produced
bare-except-chore · rep 1 · accepted · 217→9421 tok · $0.0031
no patch produced
bare-except-chore · rep 1 · accepted · 217→8102 tok · $0.0027
no patch produced
bare-except-chore · rep 2 · patch_apply_failed · 217→5949 tok · $0.0020
no patch produced
bare-except-chore · rep 2 · patch_apply_failed · 217→7301 tok · $0.0024
no patch produced
basic-function-off-by-one · rep 0 · accepted · 202→1970 tok · $0.0007
no patch produced
basic-function-off-by-one · rep 0 · accepted · 202→348 tok · $0.00013
no patch produced
basic-function-off-by-one · rep 0 · accepted · 202→1501 tok · $0.0005
no patch produced
basic-function-off-by-one · rep 1 · accepted · 202→1628 tok · $0.0005
no patch produced
basic-function-off-by-one · rep 1 · accepted · 202→1940 tok · $0.0007
no patch produced
basic-function-off-by-one · rep 1 · accepted · 202→1516 tok · $0.0005
no patch produced
basic-function-off-by-one · rep 2 · accepted · 202→1401 tok · $0.00047
no patch produced
basic-function-off-by-one · rep 2 · accepted · 202→1079 tok · $0.00037
no patch produced
config-noop-chore · rep 0 · accepted · 205→220 tok · $0.000085
no patch produced
config-noop-chore · rep 0 · accepted · 205→249 tok · $0.000094
no patch produced
config-noop-chore · rep 0 · accepted · 205→233 tok · $0.000089
no patch produced
config-noop-chore · rep 1 · accepted · 205→251 tok · $0.000095
no patch produced
config-noop-chore · rep 1 · accepted · 205→236 tok · $0.000090
no patch produced
config-noop-chore · rep 1 · accepted · 205→250 tok · $0.000095
no patch produced
config-noop-chore · rep 2 · accepted · 205→249 tok · $0.000094
no patch produced
config-noop-chore · rep 2 · patch_apply_failed · 205→4903 tok · $0.0016
no patch produced
debug-cruft-chore · rep 0 · accepted · 215→3536 tok · $0.0012
no patch produced
debug-cruft-chore · rep 0 · accepted · 215→1604 tok · $0.0005
no patch produced
debug-cruft-chore · rep 0 · accepted · 215→1591 tok · $0.0005
no patch produced
debug-cruft-chore · rep 1 · accepted · 215→2135 tok · $0.0007
no patch produced
debug-cruft-chore · rep 1 · accepted · 215→1737 tok · $0.0006
no patch produced
debug-cruft-chore · rep 1 · accepted · 215→1690 tok · $0.0006
no patch produced
debug-cruft-chore · rep 2 · accepted · 215→2070 tok · $0.0007
no patch produced
debug-cruft-chore · rep 2 · accepted · 215→1546 tok · $0.0005
no patch produced
dependency-conflict-chore · rep 0 · accepted · 299→3049 tok · $0.0010
no patch produced
dependency-conflict-chore · rep 0 · accepted · 299→4244 tok · $0.0014
no patch produced
dependency-conflict-chore · rep 1 · accepted · 299→4289 tok · $0.0014
no patch produced
dependency-conflict-chore · rep 1 · patch_apply_failed · 299→2797 tok · $0.0009
no patch produced
dependency-conflict-chore · rep 2 · accepted · 299→1568 tok · $0.0005
no patch produced
dependency-conflict-chore · rep 2 · accepted · 299→1962 tok · $0.0007
no patch produced
dependency-existing-retry-chore · rep 0 · existing_tests_failed, hidden_tests_failed, smoke_failed · 315→6441 tok · $0.0021
no patch produced
dependency-existing-retry-chore · rep 1 · existing_tests_failed, hidden_tests_failed, smoke_failed · 315→4424 tok · $0.0015
no patch produced
dependency-existing-retry-chore · rep 2 · existing_tests_failed, hidden_tests_failed, smoke_failed · 315→8253 tok · $0.0027
no patch produced
dependency-stdlib-query-chore · rep 0 · patch_apply_failed · 319→11394 tok · $0.0038
no patch produced
dependency-stdlib-query-chore · rep 1 · accepted · 319→9665 tok · $0.0032
no patch produced
dependency-stdlib-query-chore · rep 2 · patch_apply_failed · 319→2668 tok · $0.0009
no patch produced
docstring-chore · rep 0 · accepted · 168→4539 tok · $0.0015
no patch produced
docstring-chore · rep 0 · accepted · 168→2817 tok · $0.0009
no patch produced
docstring-chore · rep 0 · accepted · 168→4175 tok · $0.0014
no patch produced
docstring-chore · rep 1 · accepted · 168→3538 tok · $0.0012
no patch produced
docstring-chore · rep 1 · accepted · 168→3660 tok · $0.0012
no patch produced
docstring-chore · rep 1 · accepted · 168→4655 tok · $0.0015
no patch produced
docstring-chore · rep 2 · accepted · 168→4558 tok · $0.0015
no patch produced
docstring-chore · rep 2 · accepted · 168→3463 tok · $0.0012
no patch produced
error-message-chore · rep 0 · accepted · 254→6212 tok · $0.0021
no patch produced
error-message-chore · rep 0 · accepted · 254→4575 tok · $0.0015
no patch produced
error-message-chore · rep 0 · accepted · 254→4569 tok · $0.0015
no patch produced
error-message-chore · rep 1 · accepted · 254→4279 tok · $0.0014
no patch produced
error-message-chore · rep 1 · accepted · 254→5239 tok · $0.0017
no patch produced
error-message-chore · rep 1 · accepted · 254→6325 tok · $0.0021
no patch produced
error-message-chore · rep 2 · accepted · 254→2635 tok · $0.0009
no patch produced
error-message-chore · rep 2 · accepted · 254→6201 tok · $0.0021
no patch produced
generated-cli-help-chore · rep 0 · accepted · 223→6303 tok · $0.0021
no patch produced
generated-cli-help-chore · rep 1 · accepted · 223→4142 tok · $0.0014
no patch produced
generated-cli-help-chore · rep 2 · accepted · 223→4368 tok · $0.0015
no patch produced
generated-client-field-chore · rep 0 · accepted · 275→4676 tok · $0.0016
no patch produced
generated-client-field-chore · rep 1 · accepted · 275→6074 tok · $0.0020
no patch produced
generated-client-field-chore · rep 2 · accepted · 275→1089 tok · $0.00038
no patch produced
nextflow-orchestration-chore · rep 0 · patch_apply_failed · 484→9838 tok · $0.0033
no patch produced
nextflow-orchestration-chore · rep 0 · patch_apply_failed · 484→10962 tok · $0.0036
no patch produced
nextflow-orchestration-chore · rep 1 · forbidden_pattern, hidden_tests_failed, smoke_failed · 484→10938 tok · $0.0036
no patch produced
nextflow-orchestration-chore · rep 1 · forbidden_pattern, hidden_tests_failed, smoke_failed · 484→11640 tok · $0.0039
no patch produced
nextflow-orchestration-chore · rep 2 · patch_apply_failed · 484→10102 tok · $0.0034
no patch produced
nextflow-orchestration-chore · rep 2 · patch_apply_failed · 484→9286 tok · $0.0031
no patch produced
project-manager-chore · rep 0 · accepted · 301→5739 tok · $0.0019
no patch produced
project-manager-chore · rep 0 · patch_apply_failed · 301→4417 tok · $0.0015
no patch produced
project-manager-chore · rep 0 · patch_apply_failed · 301→1957 tok · $0.0007
no patch produced
project-manager-chore · rep 1 · accepted · 301→7318 tok · $0.0024
no patch produced
project-manager-chore · rep 1 · accepted · 301→5876 tok · $0.0020
no patch produced
project-manager-chore · rep 1 · accepted · 301→8691 tok · $0.0029
no patch produced
project-manager-chore · rep 2 · patch_apply_failed · 301→9565 tok · $0.0032
no patch produced
project-manager-chore · rep 2 · patch_apply_failed · 301→6440 tok · $0.0021
no patch produced
pyproject-edit-chore · rep 0 · accepted · 254→347 tok · $0.00013
no patch produced
pyproject-edit-chore · rep 0 · accepted · 254→256 tok · $0.00010
no patch produced
pyproject-edit-chore · rep 1 · accepted · 254→255 tok · $0.000099
no patch produced
pyproject-edit-chore · rep 1 · accepted · 254→1955 tok · $0.0007
no patch produced
pyproject-edit-chore · rep 2 · accepted · 254→218 tok · $0.000087
no patch produced
pyproject-edit-chore · rep 2 · accepted · 254→260 tok · $0.00010
no patch produced
tiny-python-type-chore · rep 0 · patch_apply_failed · 252→1834 tok · $0.0006
no patch produced
tiny-python-type-chore · rep 0 · patch_apply_failed · 185→2835 tok · $0.0009
no patch produced
tiny-python-type-chore · rep 0 · accepted · 252→3713 tok · $0.0012
no patch produced
tiny-python-type-chore · rep 1 · accepted · 252→4816 tok · $0.0016
no patch produced
tiny-python-type-chore · rep 1 · patch_apply_failed · 185→3203 tok · $0.0011
no patch produced
tiny-python-type-chore · rep 1 · accepted · 252→2684 tok · $0.0009
no patch produced
tiny-python-type-chore · rep 2 · patch_apply_failed · 252→2816 tok · $0.0009
no patch produced
tiny-python-type-chore · rep 2 · accepted · 252→2280 tok · $0.0008
no patch produced
type-annotation-chore · rep 0 · accepted · 205→2468 tok · $0.0008
no patch produced
type-annotation-chore · rep 0 · accepted · 205→682 tok · $0.00024
no patch produced
type-annotation-chore · rep 0 · accepted · 205→1946 tok · $0.0007
no patch produced
type-annotation-chore · rep 1 · accepted · 205→2412 tok · $0.0008
no patch produced
type-annotation-chore · rep 1 · accepted · 205→808 tok · $0.00028
no patch produced
type-annotation-chore · rep 1 · accepted · 205→2565 tok · $0.0009
no patch produced
type-annotation-chore · rep 2 · accepted · 205→555 tok · $0.00020
no patch produced
type-annotation-chore · rep 2 · accepted · 205→719 tok · $0.00025
no patch produced
version-hallucination-chore · rep 0 · accepted · 244→392 tok · $0.00014
no patch produced
version-hallucination-chore · rep 0 · accepted · 244→2408 tok · $0.0008
no patch produced
version-hallucination-chore · rep 1 · accepted · 244→2294 tok · $0.0008
no patch produced
version-hallucination-chore · rep 1 · accepted · 244→3474 tok · $0.0012
no patch produced
version-hallucination-chore · rep 2 · accepted · 244→1468 tok · $0.00050
no patch produced
version-hallucination-chore · rep 2 · accepted · 244→2169 tok · $0.0007
no patch produced