Benchmark report · Tencent HY LLM Frontier · July 2026

Long-Horizon Terminal-Bench

Measuring the progress agents can sustain, not just what they can finish.

Most agent benchmarks end in minutes. Real terminal work may not. Long-Horizon Terminal-Bench drops an agent into a Docker container with a goal that takes hundreds of dependent actions — build a system from scratch, migrate a real framework across breaking changes, play a game move by move — then grades it with a hidden, fake-proof verifier that pays continuous partial credit. We ran 18 frontier models through the same harness. None passes even a third of the tasks.

Long-Horizon Terminal-Bench (LHTB)

46 tasks · 9 categories · 18 models ·

The loop that has to hold for ninety minutes. The agent types commands into a live containerized environment and reads text observations back — hundreds of times. Only when it stops does a hidden verifier replay the evidence and pay a graded reward. Progress can't be faked; only real work counts.

In one minute

Long-Horizon Terminal-Bench (LHTB) is a suite of 46 hard, reproducible terminal tasks across 9 categories, designed to resist memorization, shortcutting, and reward hacking. Every task pays continuous partial credit instead of binary pass/fail.
We evaluated 18 frontier models under one identical harness (Terminus-2), one container per task, a 90-minute budget, and hidden verifiers.
The frontier tops out below half credit: the best model (Grok 4.5) averages 0.51, and also posts the best solve count, 13 of 46. 29 of 46 tasks have never been passed by any model.
Price is not performance: MiniMax M3 scores 0.39 at ~$6 per task — ahead of GPT-5.4 at $28 and other pricier models. The signal lives in the partial credit, where binary metrics see almost no difference.

How long is “long-horizon”?

Running all 46 tasks once takes each model 53–71 hours of wall-clock time. A single task averages 69–93 minutes and ~120–320 agent steps — the loop the model has to hold together, without losing the thread, before a hidden verifier ever sees the result.

The scale of one run, per model

Averaged over the 46-task suite · one 90-minute attempt each

Even the most economical agents spend well over two full days of compute-time to work through the suite once, and hundreds of dependent steps on a single task.

Part 01

Agents break at the horizon, not at the task

Ask a frontier model to fix a failing test and it usually can. Ask it to spend an afternoon bringing a scientific codebase back to numerical parity — running simulations, reading regressions, adjusting, re-running — and something different happens. The model does not lack the knowledge. It loses the thread: state drifts, earlier decisions are forgotten, exploration turns into looping, and the budget burns down with the goal half-reached.

Existing terminal benchmarks rarely see this failure mode because their tasks can be short. A task that ends after a dozen actions measures whether an agent can start work. It says nothing about whether the agent can sustain it — carry state across hundreds of steps, recover from dead ends, and know when it is actually done.

LHTB was built to measure exactly this regime. Three design axes run through every task: (1) long horizon — solutions need hundreds of dependent actions and sustained state; (2) resistance — hidden verifiers, deterministic seeds, and replay-based grading make memorization, shortcutting, and reward hacking unprofitable; and (3) graded reward — a continuous score for partial progress, so the benchmark ranks models even where nobody passes.

Part 02

The benchmark: 46 tasks, 9 categories

Every task follows the same contract: a Docker container, an instruction file, a live environment, and a hidden test suite that grades the outcome. The 46 tasks span nine categories — software & reverse engineering, scientific computing, earth/climate & energy, multimodal & imaging, research reproduction, systems & security, professional (APEX) workflows, and two that demand sustained interaction rather than a single artifact: interactive games played turn by turn, and logic & constraint puzzles solved by search and reasoning.

Scale

46 tasks · 9 categories

From framework migrations and chip-design signoff to seismic-regression audits and playing 2048.

Grading

Hidden, fake-proof verifiers

Deterministic replay, seeded environments, held-out answer keys. Claimed progress does not count — replayed progress does.

Reward

Continuous, 0 → 1

Banded or proportional partial credit per task; a task counts as solved at reward ≥ 0.95.

What the 46 tasks cover

Tasks per category

Deliberately wide. Software & reverse engineering leads with seven tasks; scientific computing, earth/climate, and multimodal analysis each contribute six of rebuilding and evidence extraction; and interactive games and logic puzzles keep sustained, turn-by-turn play in the mix. Even the largest category is only 7 of 46, so no single skill dominates the mean.

Part 03

Anatomy of a task

The clearest way to see the design is through one concrete task. In unison-paper-reproduction, the agent reproduces an experiment from the UNISON paper without being given the original implementation.

Task · Research reproduction

Reproducing the UNISON fat-tree experiment

From the paper alone, build a parallel simulation pipeline, run the fat-tree MTP experiment (k=4, 4 threads, seed 7), and match the reference metrics — 232 flows, 6,254,916 events, 14.46 Gbps — within tolerance.

Environment: A skeleton project in /app; the agent must implement topology generation, LP partitioning, deterministic execution, and scheduling.
Deliverables: A run-fat-tree CLI plus unison_report.json and summary.csv with LP counts, scheduler stats, and a deterministic checksum.
Verifier: Six hidden checks cover metric fidelity, non-trivial LP partitioning and scheduling, same-seed determinism, and plausible shifts under hidden seeds.
Reward: Fraction of checks passed; a deterministic pipeline that misses the target numbers still earns partial credit.

Layout hover to preview · click to pin →

actual task file

Part 04

Evaluation Setup

Comparisons are only meaningful if the scaffolding is identical. Every model runs through the same Terminus-2 harness — same prompts, same parser, same summarization policy — differing only in the API endpoint behind it. No model-specific tools, no bespoke agents, no retries of bad runs.

Harness

Terminus-2, identical for all

JSON action parser, proactive context summarization, full terminal-session recording.

Budget

90 min per task

One attempt. A timeout keeps its partial credit — whatever the verifier can replay at the deadline.

Metrics

Mean reward · solved @ ≥0.95

Errors score 0. Mean reward over all 46 tasks is the headline number; solve count is the sparser, harsher view.

Part 05

Results

Result 1 The frontier tops out below half credit

Averaged over all 46 tasks, the best model earns less than half the available reward. Grok 4.5 narrowly leads a trio of Anthropic models, the GPT-5 series (5.6 and 5.5) follows, and a tight pack of strong open-weight and Chinese frontier models lands between 0.25 and 0.39. The spread from first to last is nearly 2.5× — this benchmark is far from saturated.

Leaderboard — 46 tasks

Mean reward over 46 tasks · solved = reward ≥ 0.95 · one identical harness

Rank by

Pass threshold · R ≥ 0.95

0.900.951.00

Partial credit is what keeps the ranking alive. On mean reward, Grok 4.5 leads (0.505) and the field spreads across the chart. Switch Rank by to pass rate and drag the pass threshold up to a perfect reward of R = 1.0 — the ranking reshuffles and the bars collapse: ten of the models earn real partial credit yet solve zero tasks outright. The signal lives in the partial reward.

Result 2 Partial credit shows the signal

Look at all 782 model-task runs at once and the case for graded rewards makes itself. Only 7% of runs cross the solve threshold — under binary scoring, 93% of the benchmark would be indistinguishable zeros. The continuous reward spreads that mass out: more than half of all runs land in the low-partial band (0–0.25), where models differ by how far they get before losing the thread.

Where 782 runs land

% of all model × task runs, by reward band

The unsolved frontier

Of 46 tasks, how many has any model solved?

29 of 46 tasks have never been solved by any model. And on 9.2% of runs the agent earned exactly nothing. A benchmark scored pass/fail would compress this landscape to near-zero everywhere; partial credit is what keeps the ranking informative.

The hardest tasks demand sustained quantitative feedback loops — run, measure, adjust, re-run — or evidence extraction from non-text modalities.

Hardest tasks	Domain	Mean (18 models)	Best single run

The open frontier. On robotics-slam-benchmark-repair the best model earned 0.03. At the other end, spot-scheduler-traces (mean 0.96) and nbody-accel-iterative (0.93) show the field can execute long tasks when the feedback signal is legible.

Result 3 Price is not performance

The cost axis tells its own story. Per-task spend ranges nearly 30× (from $2.5 (Hy3) to $73 (Claude Fable 5)) while mean reward spans only 2.5×. The twist: Grok 4.5 posts the top mean reward at just ~$11 per task. The Anthropic models pay the largest budgets ($38–73 per task) for their scores, while a cluster around $4–12 per task delivers 60–80% of the top score at a fraction of the cost.

Mean reward vs. cost per task

Cost on a log scale · up-and-left is better

The efficient frontier moved. Grok 4.5 lands top-left — the best mean reward (0.505) at just $11.19 per task, top score and low cost at once. Below it, MiniMax M3 (0.385 at $6.13) and Qwen3.6 Plus (0.313 at $4.47) remain the value picks, while the Anthropic models pay $38–73 for comparable or lower scores.

Part 06

Takeaways

Long-horizon completion is the real bottleneck. Across the models, runs average 231 steps, 9.9M tokens, and 85 minutes, yet the best model still solves only 7 of 46 tasks at reward=1.0.
Most failures are not immediate mistakes but unfinished work: 79% of unresolved runs time out while the agent is still actively making progress.
Dense rewards are essential at this difficulty level. Under a strict perfect-score threshold, ten of the fifteen models solve zero tasks, even though many of their runs make substantial partial progress.