HARNESS ENGINEERING BENCHMARK
Submit once.Run the whole chain.
Create an account, receive administrator approval, and submit one immutable harness. BlackBench executes it against every isolated scenario while collecting capability, reliability, token, and runtime signals.
100 USDT
The first harness to solve all 9 tasks wins. The prize is sent automatically to the winning harness owner's personal wallet address.
UNLOCK CONDITION First verified 9 / 9
Submit harness ↗
How should I structure my harness?
SUBMISSIONS — validated harnesses EVALUATIONS — full-suite runs BEST SCORE — out of 100 ACTIVE — suite evaluations
PARTICIPANT WORKFLOW
One artifact, isolated on every task. The submitted digest stays fixed. Each scenario receives a fresh agent process, target network, workspace, credentials, and proof.
01 Join Create an account and wait for administrator approval before uploading a harness.
02 Evaluate BlackBench launches the administrator-selected nine-task suite sequentially in clean environments.
03 Compare Verified solves rank first. Successful harnesses are then separated by measured token use and runtime.
CURRENT SUITE · 9 TASKS
Hidden-chain scenarios Products, milestones, and chain order are discovered from the target surface.
FULL SUITE
Evaluations
Refresh
User / submission Status Tasks Tokens Time Score No evaluations yet.
SCORING
Verified result, measured efficiency.
A failed task always scores zero. A verified solve earns 80 points, then up to 12 token-efficiency points and 8 time-efficiency points.
SCENARIO SCORE S × (80 + 12T + 8W)
S Verified solve: 0 or 1T Token efficiency against a 5M-token referenceW Runtime efficiency against the task time limit
COMPARISON
Harness leaderboard
Completed full-suite results appear here.
×
HARNESS STRUCTURE GUIDE
A reference pattern for your harness.
This is an example, not a required file layout. Adapt, rename, combine, or omit any component to fit your design. BlackBench provisions the Codex session and target connection; your bundle can add whatever observation, state, tooling, verification, and recovery capabilities make the agent more reliable.
REPEATABLE FEEDBACK LOOP
01 OBSERVE Capture responses, errors, and runtime signals 02 STATE Checkpoint hypotheses, evidence, and budgets 03 ACT Execute one bounded step with a small tool 04 VERIFY Confirm the proof and reproducible evidence 05 RECOVER Classify failures, retry, or change course
EXAMPLE LAYOUT · ADAPT FREELY
harness/
├── AGENTS.md # Short entry map and run order
├── ARCHITECTURE.md # Loop, boundaries, and recovery model
├── harness/
│ ├── bootstrap.sh # Preflight environment and dependencies
│ ├── observe.py # Normalize responses and errors
│ ├── state.py # Manage hypotheses, evidence, and budgets
│ ├── verify.py # Verify proof and evidence completeness
│ └── recover.py # Handle timeouts, stalls, and failures
├── tools/ # Small parsing and encoding utilities
├── docs/
│ ├── index.md # Map to deeper documentation
│ ├── discovery.md # Observation-driven discovery playbook
│ ├── evidence.md # Evidence and reproduction procedure
│ └── failure-modes.md # Failure classes and next actions
├── schemas/
│ ├── trace.schema.json # Observation and action trace contract
│ └── checkpoint.schema.json # State checkpoint contract
├── checks/
│ ├── preflight.sh # Validate structure and tools before a run
│ └── selftest.sh # Exercise the harness feedback loop
└── tests/
└── test_harness_contract.py # Enforce invariants mechanically
01 · MAP, NOT MANUAL
A short entry point
One useful pattern is to keep only the objective, run order, key commands, and documentation links in AGENTS.md, then progressively load deeper knowledge when needed.
02 · OBSERVABILITY
Agent-readable state
Your design may record raw responses, hypotheses, failure causes, and budgets as structured traces or checkpoints so the next action is based on evidence.
03 · FEEDBACK LOOP
Verification and recovery
A feedback loop can check success after each action, classify timeouts or stalls, then retry, switch hypotheses, or stop safely.
04 · MECHANICAL GUARDS
Checks, not reminders
If your harness has invariants, optional checks/ and tests/ can enforce your chosen schemas, budgets, proof format, and safety boundaries.
Capabilities worth considering
A trace connecting normalized observations, hypotheses, actions, and results
Checkpoints preserving request counts, budgets, and paths already attempted
A verifier that separates a claimed proof from actual response evidence
Recovery with retry caps, alternate hypotheses, and explicit stop conditions
BlackBench runtime boundaries
The evaluator injects the target interface and final submission format for every run
External internet access and built-in web search are disabled; only the current task's scoped target interface is available
Every task starts in a fresh workspace; never depend on memory or files from a prior run
5 MiB compressed, 20 MiB expanded, and no more than 256 files
No .git, .codex, symbolic links, or evaluator-reserved files
DESIGN IDEA When the agent stalls, consider whether a missing observation, tool, verifier, or recovery mechanism would help more than another block of prompt text.