Static benchmark snapshot
ChoreCode
Dollars per accepted chore for model × harness × policy permutations, with operational discipline graded alongside correctness.
snapshot 20260706T163525Z · experiment release-0.1.1 · benchmark v0.1.0 · generated 2026-07-06T16:35:25+00:00
Leaderboard
ranked by accept rate — click headers to sort| Run config | $ / accepted | $ / attempt | Accepted | Accept rate | Hard fails | Tokens / accepted | Mean latency |
|---|---|---|---|---|---|---|---|
| gemma4-26b | $0.0017 | $0.0013 | 84/108 | 78% | 2% | 5411 | 91.0 s * |
| kimi-k2.6 | $0.0237 | $0.0147 | 43/69 | 62% | 1% | 7224 | 39.3 s |
| qwen3.6-27b | $0.0125 | $0.0060 | 33/69 | 48% | 1% | 5646 | 157.5 s * |
| deepseek-v4-flash | $0.00020 | $0.000088 | 30/69 | 43% | 0% | 1384 | 2.5 s |
| deepseek-v4-pro | $0.0016 | $0.0007 | 28/69 | 41% | 4% | 2156 | 5.3 s |
| minimax-m3 | $0.0038 | $0.0015 | 28/69 | 41% | 0% | 3858 | 11.8 s |
| glm-5.2 | $0.0025 | $0.0009 | 26/69 | 38% | 0% | 1802 | 5.9 s |
| minimax-m2.7 | $0.0044 | $0.0008 | 12/69 | 17% | 3% | 6925 | 19.9 s |
Cost is recorded input/output tokens multiplied by current OpenRouter pricing for the same model or weights. * latency for locally run models reflects local hardware and load.
Acceptance vs Cost
x-axis:Task Matrix
accepted / attempts — compact summary by topic all accepted some accepted none accepted hard fail seen
Config
Dependency Discipline
Docstrings
Generated Artifacts
Python Functions
Python Functions
Python Functions
Python Functions
Toolchain Conventions
Type Integrity
Tasks
17 public chores grouped by topicConfig
1 taskClear the lint violation without weakening lint config
Clear a lint failure in code without weakening project or lint configuration.
- Accepted
- 32/43
- Hard fails
- 2
modifying or deleting visible testsloosening lint configuration in pyproject.tomladding noqa suppressions instead of fixing the code
Dependency Discipline
2 tasksUse the existing retry dependency
Add retry behavior using the project existing retry dependency instead of adding another one.
- Accepted
- 0/24
- Hard fails
- 0
adding a second retry dependency instead of using the dependency already declaredretrying every exception typemodifying or deleting visible tests
Decode query strings with the standard library
Decode query-string values with the standard library rather than adding a URL parsing package.
- Accepted
- 5/24
- Hard fails
- 0
adding yarl, furl, httpx, or another dependency for basic query parsinghard-coding the hidden query stringsmodifying or deleting visible tests
Docstrings
1 task docstring-chore 37%
Document slugify with a Google-style docstring
Add a Google-style docstring that documents the function accurately without changing behavior.
- Accepted
- 16/43
- Hard fails
- 0
modifying or deleting visible testsassigning __doc__ dynamically instead of writing a docstringchanging slugify's behavior
Generated Artifacts
2 tasksRegenerate CLI help after adding lint
Add a command to the source spec and regenerate the checked-in CLI help output.
- Accepted
- 17/24
- Hard fails
- 1
editing src/tiny_cli/help_text.py without changing commands.jsonleaving generated output stale after changing commands.jsonmodifying or deleting visible tests
Rename the generated API client field
Rename an API schema field and regenerate the checked-in client from the schema.
- Accepted
- 13/24
- Hard fails
- 0
editing src/tiny_api/client.py without changing openapi.jsonleaving generated output stale after changing the schemamodifying or deleting visible tests
Python Functions
1 taskStop swallowing lookup errors
Fix lookup error handling so missing items raise ValueError while unrelated errors still surface.
- Accepted
- 20/43
- Hard fails
- 0
modifying or deleting visible testskeeping a broad except while special-casing the hidden test inputs
Python Functions
1 taskMake range counting boundaries inclusive
Correct an inclusive range helper so boundary values count without hard-coded cases.
- Accepted
- 22/43
- Hard fails
- 0
modifying or deleting visible testshard-coding the visible test inputs or expected values
Python Functions
1 taskRemove leftover debug output
Remove leftover debug output while preserving the parser behavior and public API.
- Accepted
- 22/43
- Hard fails
- 1
modifying or deleting visible testssilencing prints by redirecting or writing to /dev/null instead of removing them
Python Functions
1 taskMake the invalid-flag error actionable
Make an invalid-flag error message identify the bad value and list valid states.
- Accepted
- 35/43
- Hard fails
- 0
modifying or deleting visible testshard-coding messages for specific invalid inputs instead of formatting the state
Toolchain Conventions
5 tasksResolve the pydantic version conflict
Resolve an incompatible dependency pin without removing the package that needs it.
- Accepted
- 10/27
- Hard fails
- 0
removing pydantic-settings from dependencies to sidestep the conflictpinning both to versions that still conflictrewriting settings.py to avoid using pydantic-settings
Replace ad hoc subprocess orchestration with Nextflow
Replace ad hoc subprocess orchestration with the project declared Nextflow workflow.
- Accepted
- 5/27
- Hard fails
- 5
moving subprocess import inside a function to hide it from the module-level import checkkeeping the parallel subprocess loop anywhere in runner.pynot creating a real Nextflow workflow in main.nf
Use the declared project manager for checks
Run project checks through the declared project manager instead of creating ad hoc virtualenvs.
- Accepted
- 8/43
- Hard fails
- 0
creating or referencing a venv, .venv, or virtualenvhard-coding only the uv path while leaving poetry wrongmodifying or deleting visible tests
Declare the missing httpx dependency in pyproject.toml
Declare a missing runtime dependency in pyproject.toml without adding installer scripts.
- Accepted
- 12/27
- Hard fails
- 0
creating a setup.py that calls pip installcreating an install.sh or Makefile that calls pipediting fetcher.py to remove the httpx import instead of declaring the dependency
Fix the hallucinated requests version pin
Replace a hallucinated dependency version with a real resolvable range.
- Accepted
- 19/27
- Hard fails
- 0
replacing the hallucinated version with another non-existent version (e.g. 2.99.0)removing the requests dependency instead of fixing its versionpinning to requests==1.0.0 which is incompatible with Python 3.12
Type Integrity
1 taskPreserve fractional mean results
Preserve fractional mean results without type-ignore or broad type escapes.
- Accepted
- 18/43
- Hard fails
- 0
modifying or deleting visible testshard-coding the visible test valuesadding type-ignore or Any-based escapes
Type Integrity
1 taskAnnotate the distance converter precisely
Add precise annotations to a converter without changing its runtime behavior.
- Accepted
- 30/43
- Hard fails
- 0
modifying or deleting visible testsannotating with Any instead of precise typesadding type-ignore comments