Submitting to the LHTB leaderboard

The Long-Horizon-Terminal-Bench (LHTB) leaderboard follows the same submission flow as Terminal-Bench 2: run the full 46-task suite with five trials per task under one shared harness, then open a pull request against this repo with your job directories. A bot validates the submission and a maintainer reviews the trajectories against the hidden verifiers before it lands on the board.

Repo: IntelligenceLab/LHTB-leaderboard (Hugging Face Dataset)

1. Run all 46 tasks, five trials each

Use the reference terminus-2 harness and a 90-minute budget per task. -k 5 runs each task five times.

Evaluate a model (reference harness):

harbor run -d long-horizon-terminal-bench \
  -a terminus-2 -m "your-provider/your-model" -k 5

Bring your own agent:

harbor run -d long-horizon-terminal-bench \
  --agent-import-path "path.to.agent:YourAgent" -k 5

This produces a job directory containing a config.json and one folder per trial, each with a result.json and the run artifacts (logs, terminal recordings, etc.).

2. Fork this repo and add your jobs

Fork IntelligenceLab/LHTB-leaderboard, then drop your job folder(s) under:

submissions/long-horizon-terminal-bench/1.0/<agent>__<model>/

Alongside a metadata.yaml. Full layout:

submissions/long-horizon-terminal-bench/1.0/
  terminus-2__your-model/
    metadata.yaml
    <job-folder>/
      config.json
      <trial-1>/result.json
      <trial-2>/result.json
      ...              # >= 5 trials/task, with artifacts

metadata.yaml:

agent: terminus-2
model: your-provider/your-model
organization: Your Org
repo: https://github.com/your/agent   # optional
docs: https://.../model-card          # optional

3. Open a pull request

Push your fork and open a PR against IntelligenceLab/LHTB-leaderboard.

• The validation bot auto-checks the submission and comments on any errors.
• Once it passes, a maintainer reviews the trajectories and merges.
• Your run is then imported to the board with a verified badge.

Validation rules

A submission is valid only if all of the following hold:

• Five trials per task (-k 5).
• timeout_multiplier = 1.0 — no agent or verifier timeout overrides.
• No CPU / memory / storage overrides.
• Every trial has a valid result.json with run artifacts.
• Agents may not access the LHTB website or repository (reward hacking).
• Errored trials score 0.

Scores are mean reward over the 46-task suite (errors = 0). A task counts as solved at reward ≥ 0.95; the strict binary pass rate uses reward = 1.0.