Benchmark report · Tencent HY LLM Frontier · July 2026
Long-Horizon Terminal-Bench
Measuring the progress agents can sustain, not just what they can finish.
Most agent benchmarks end in minutes. Real terminal work may not. Long-Horizon Terminal-Bench drops an agent into a Docker container with a goal that takes hundreds of dependent actions — build a system from scratch, migrate a real framework across breaking changes, play a game move by move — then grades it with a hidden, fake-proof verifier that pays continuous partial credit. We ran 18 frontier models through the same harness. None passes even a third of the tasks.
In one minute
- Long-Horizon Terminal-Bench (LHTB) is a suite of 46 hard, reproducible terminal tasks across 9 categories, designed to resist memorization, shortcutting, and reward hacking. Every task pays continuous partial credit instead of binary pass/fail.
- We evaluated 17 frontier models under one identical harness (Terminus-2), one container per task, a 90-minute budget, and hidden verifiers.
- The frontier tops out below half credit: the best model (Claude Opus 4.8) averages 0.49, and the best solve count is 12 of 46 (Claude Fable 5). 29 of 46 tasks have never been passed by any model.
- Price is not performance: MiniMax M3 scores 0.39 at ~$6 per task — ahead of GPT-5.4 at $28 and other pricier models. The signal lives in the partial credit, where binary metrics see almost no difference.
How long is “long-horizon”?
Running all 46 tasks once takes each model 53–71 hours of wall-clock time. A single task averages 69–93 minutes and ~120–320 agent steps — the loop the model has to hold together, without losing the thread, before a hidden verifier ever sees the result.
The scale of one run, per model
Averaged over the 46-task suite · one 90-minute attempt each
Part 01
Agents break at the horizon, not at the task
Ask a frontier model to fix a failing test and it usually can. Ask it to spend an afternoon bringing a scientific codebase back to numerical parity — running simulations, reading regressions, adjusting, re-running — and something different happens. The model does not lack the knowledge. It loses the thread: state drifts, earlier decisions are forgotten, exploration turns into looping, and the budget burns down with the goal half-reached.
Existing terminal benchmarks rarely see this failure mode because their tasks can be short. A task that ends after a dozen actions measures whether an agent can start work. It says nothing about whether the agent can sustain it — carry state across hundreds of steps, recover from dead ends, and know when it is actually done.
LHTB was built to measure exactly this regime. Three design axes run through every task: (1) long horizon — solutions need hundreds of dependent actions and sustained state; (2) resistance — hidden verifiers, deterministic seeds, and replay-based grading make memorization, shortcutting, and reward hacking unprofitable; and (3) graded reward — a continuous score for partial progress, so the benchmark ranks models even where nobody passes.
Part 02
The benchmark: 46 tasks, 9 categories
Every task follows the same contract: a Docker container, an instruction file, a live environment, and a hidden test suite that grades the outcome. The 46 tasks span nine categories — software & reverse engineering, scientific computing, earth/climate & energy, multimodal & imaging, research reproduction, systems & security, professional (APEX) workflows, and two that demand sustained interaction rather than a single artifact: interactive games played turn by turn, and logic & constraint puzzles solved by search and reasoning.
Scale
46 tasks · 9 categories
From framework migrations and chip-design signoff to seismic-regression audits and playing 2048.
Grading
Hidden, fake-proof verifiers
Deterministic replay, seeded environments, held-out answer keys. Claimed progress does not count — replayed progress does.
Reward
Continuous, 0 → 1
Banded or proportional partial credit per task; a task counts as solved at reward ≥ 0.95.
What the 46 tasks cover
Tasks per category
Part 03
Anatomy of a task
The clearest way to see the design is through one concrete task. In
unison-paper-reproduction, the agent reproduces an experiment from the
UNISON
paper without being given the
original implementation.
Reproducing the UNISON fat-tree experiment
From the paper alone, build a parallel simulation pipeline, run the fat-tree MTP experiment (k=4, 4 threads, seed 7), and match the reference metrics — 232 flows, 6,254,916 events, 14.46 Gbps — within tolerance.
- Environment
- A skeleton project in
/app; the agent must implement topology generation, LP partitioning, deterministic execution, and scheduling. - Deliverables
- A
run-fat-treeCLI plusunison_report.jsonandsummary.csvwith LP counts, scheduler stats, and a deterministic checksum. - Verifier
- Six hidden checks cover metric fidelity, non-trivial LP partitioning and scheduling, same-seed determinism, and plausible shifts under hidden seeds.
- Reward
- Fraction of checks passed; a deterministic pipeline that misses the target numbers still earns partial credit.
instruction.md
environment/ · skeleton app
tests/ · hidden verifier
task.toml
Part 04
Evaluation Setup
Comparisons are only meaningful if the scaffolding is identical. Every model runs through the same Terminus-2 harness — same prompts, same parser, same summarization policy — differing only in the API endpoint behind it. No model-specific tools, no bespoke agents, no retries of bad runs.
Harness
Terminus-2, identical for all
JSON action parser, proactive context summarization, full terminal-session recording.
Budget
90 min per task
One attempt. A timeout keeps its partial credit — whatever the verifier can replay at the deadline.
Metrics
Mean reward · solved @ ≥0.95
Errors score 0. Mean reward over all 46 tasks is the headline number; solve count is the sparser, harsher view.
Part 05
Results
Result 1 The frontier tops out below half credit
Averaged over all 46 tasks, the best model earns less than half the available reward. Two Anthropic models lead, GPT-5.5 follows, and a tight pack of strong open-weight and Chinese frontier models lands between 0.25 and 0.39. The spread from first to last is nearly 2.5× — this benchmark is far from saturated.
Leaderboard — mean reward over 46 tasks
Errors count as 0 · solved = reward ≥ 0.95 · one identical harness
Result 2 The signal lives in the partial credit
Look at all 782 model-task runs at once and the case for graded rewards makes itself. Only 7% of runs cross the solve threshold — under binary scoring, 93% of the benchmark would be indistinguishable zeros. The continuous reward spreads that mass out: more than half of all runs land in the low-partial band (0–0.25), where models differ by how far they get before losing the thread.
Where 782 runs land
% of all model × task runs, by reward band
The unsolved frontier
Of 46 tasks, how many has any model solved?
The hardest tasks share a shape: they demand sustained quantitative feedback loops — run, measure, adjust, re-run — or evidence extraction from non-text modalities. The easiest share the opposite shape: a crisp objective with a visible signal the agent can hill-climb.
| Hardest tasks | Domain | Mean (17 models) | Best single run |
|---|
robotics-slam-benchmark-repair the best
model earned 0.03 — seventeen frontier models, effectively shut out. At the other end,
spot-scheduler-traces (mean 0.96) and nbody-accel-iterative
(0.93) show the field can execute long tasks when the feedback signal is legible.
Result 3 Price is not performance
The cost axis tells its own story. Per-task spend ranges nearly 30× — from $2.5 (hy3) to $73 (Claude Fable 5) — while mean reward spans only 2.5×. The Anthropic models buy their lead with the largest budgets ($38–73 per task); meanwhile a cluster of models around $4–12 per task delivers 60–80% of the leader's score at roughly a sixth of the cost.
Mean reward vs. cost per task
Cost on a log scale · up-and-left is better
Part 06
What we learned
The horizon is the bottleneck, not the knowledge
Average runtimes cluster at 69–93 minutes against a 90-minute budget — most models work until the clock stops them, mid-task. The failures are rarely "didn't know how"; they are drift, looping, and lost state deep into the run. More budget helps some tasks, but the models that lead do more per step, not just more steps.
Graded rewards separate what pass/fail cannot
Kimi K2.6 solves zero tasks yet averages 0.25 — on a binary benchmark it would tie with a model that produced nothing at all. Nearly half of all runs land between 0 and 0.25, and that band is where most of today's frontier competition actually happens.
The efficient frontier is crowded at the bottom
Cost per task varies nearly 30×; capability varies 2.5×. A team that can accept a 0.39-mean agent (MiniMax M3, $6) over the 0.49-mean leader (Opus, $39) cuts inference cost about 6× — a trade many production workloads will gladly make.
Multimodal and control-loop tasks are wide open
All five multimodal tasks sit in the hardest quartile, and quantitative control-loop audits (robotics SLAM, seismic regression, epidemic inverse control) have a best score under 0.17 across every model. Whoever cracks sustained measure-adjust-rerun loops moves the whole leaderboard.
Part 07
Takeaways
- Long-horizon execution is a distinct capability — and today's frontier earns less than half credit on it.
- Hidden, replay-based verifiers with continuous rewards keep a hard benchmark informative: 29 never-solved tasks still produce a clean ranking.
- Identical-harness evaluation isolates the model. The 2.5× spread you see is model capability, not scaffolding.
- Watch the $4–12-per-task tier: it improves fastest, and it is already 60–80% of the way to the leaders at roughly a sixth of the cost.
The benchmark's most important number is not the winner's 0.49 — it is the 29 tasks nobody has solved. That is the map of where agent capability ends today, and where the next generation of models has to go.