Evals

See also eval/tasks/readme.md and spec/eval.md in the Zox repo.

Commands (README)

Command	Purpose
`zox eval run`	Fixture evals (default: mock tasks)
`zox eval private`	Hidden-grader private suite
`zox eval swe-lite`	SWE-bench Lite subset (Docker)
`zox eval terminal-bench`	Terminal-Bench 2.0 (Docker)
`zox eval archive`	Archive last eval artifacts

YAML tasks with expect.stdoutIncludes and/or expect.files.

zox eval run                          # eval/tasks/mock + mock/echo
zox eval run eval/tasks/live --model openai/gpt-4.1

--max-turns (live default 50), --timeout-ms (default 300000), --sandbox.

eval/private/<id>/: README.md prompt, repo/ start tree, hidden tests/ + grader.sh. Agent never sees hidden tests. Default k=3.

zox eval private --model openai/gpt-4.1 --k 1

Artifacts: eval/results/private/<task>/trial-N/.

Needs Docker and swebench in a venv. Real --model (not mock/*).

zox eval swe-lite --model openai/gpt-4.1 --limit 1
zox eval swe-lite --model openai/gpt-4.1 --skip-eval

Docker + Harbor. Adapter: eval/adapters/harbor/zox_agent.py.

zox eval terminal-bench --model openai/gpt-4.1 --limit 1