Terminal-Bench 3.0
Switch benchTerminal-Bench 3.0
Request dataset samplesby

Firmware Patch BackdoorSecurity & Forensics

Human baseline4h

TaxonomySecurity & ForensicsSecurity & Forensics

Start with all trials for auto-written insights, or select a single trial to read its full trajectory.

Opus-4.8claude-code

All trials8 trialsTrial 01100%Trial 02100%Trial 03100%Trial 04100%Trial 05100%Trial 060%Trial 07100%Trial 08100%

Mean reward

87.5%

All trials

Across 8 trials on this task, reward ran from 0% to 100% — 7 of 8 passed.

  • $0.73avg cost
  • 4mavg time

Summary · Opus-4.8

Opus-4.8 passed 7 of 8 trials.

Common pattern

7 of 8 trials passed on identical attempts: the capability is there, the reliability is not.

Gemini-3gemini-cli

All trials8 trialsTrial 010%Trial 020%Trial 030%Trial 040%Trial 050%Trial 060%Trial 070%Trial 080%

Mean reward

0%

All trials

Across 8 trials on this task, reward ran from 0% to 0% — 0 of 8 passed.

  • $0.03avg cost
  • 0mavg time

Summary · Gemini-3

Gemini-3 passed none of its 8 trials.

Common pattern

No trial passed. Whatever this task demands, the model misses it every time — a reproducible miss, not variance.

GPT-5.4codex

All trials8 trialsTrial 01trace0%Trial 020%Trial 030%Trial 040%Trial 050%Trial 060%Trial 070%Trial 080%

Mean reward

0%

All trials

Across 8 trials on this task, reward ran from 0% to 0% — 0 of 8 passed.

  • 39avg turns
  • $0.27avg cost
  • 1mavg time

Summary · GPT-5.4

GPT-5.4 passed none of its 8 trials.

Common pattern

No trial passed. Whatever this task demands, the model misses it every time — a reproducible miss, not variance.

Published trajectory

The drop publishes a full step record for one of the trials above — every other attempt shipped its score without its route.

GPT-5.4Trial 01Fail39 steps
The agent's last word

“The artifacts are written under [/logs/artifacts](/logs/artifacts): `cert_validate` contains the malicious patch. The patched instruction starts at offset `0x70` in `firmware_v2.bin`, accepts the legacy certificate OID `0x00badc0d`, and unlocks a path that yields the payload `exfiltrated_core_secret=RV32I_PATCH_BACKDOOR`. Files: - [/logs/artifacts/offset.txt](/logs/artifacts/offset.txt) - [/logs/artifacts/function.txt](/logs/artifacts/function.txt) - [/logs/artifacts/trigger.txt](/logs/artifacts/trigger.txt) - [/logs/artifacts/payload.txt](/logs/artifacts/payload.txt)”

The full trajectory
  1. system

    <permissions instructions> Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled. Approval policy is currently never. Do not provide the `sandbox_permissions` for any reason, commands will be rejected. </permissions instructions><skills_instructions> ## Skills A skill is a set of instructions provided through a `SKILL.md` source. Below is the list of skills that can be used. Each entry includes a name, description, and source locator. `file` locators are on the host filesystem, `environment resource` locators are owned by an execution environment, `orchestrator resource` locators are opaque non-filesystem resources, and `custom resource` locators use their provider's access mechanism. ### Available skills - imagegen: Generate or edit raster images when the task benefits from AI-created bitmap visuals such as photos, illustrations, textures, sprites, mockups, or transparent-background cutouts. Use when Codex should create a brand-new image, transform an existing image, or derive visual variants from references, and the output should be a bitmap asset rather than repo-native code or vector. Do not use when the task is better handled by editing existing SVG/vector/code-native assets, extending an established icon or logo system, or building the visual directly in HTML/CSS/canvas. (file: /tmp/codex-home/skills/.system/imagegen/SKILL.md) - openai-docs: Use when the user asks how to build with OpenAI products or APIs, asks about Codex itself or choosing Codex surfaces, needs up-to-date official documentation with citations, help choosing the latest model for a use case, latest/current/default-model prompting guidance, or model upgrade and prompt-upgrade guidance; use OpenAI docs MCP tools for non-Codex docs questions, use the Codex manual helper first for broad Codex self-knowledge, and restrict fallback browsing to official OpenAI domains.

  2. user

    <environment_context> <cwd>/app</cwd> <shell>bash</shell> <current_date>2026-07-23</current_date> <timezone>Etc/UTC</timezone> <filesystem><workspace_roots><root>/app</root></workspace_roots><permission_profile type="disabled"><file_system type="unrestricted" /></permission_profile></filesystem> </environment_context>

  3. user

    # Firmware patch backdoor Two signed RISC-V firmware images for a small device are in `/app`: - `firmware_v1.bin` — the release from day N. - `firmware_v2.bin` — the release from day N+1, signed and claiming only bugfixes. - `bootloader.elf` — the day N+1 bootloader as an ELF with symbols. The vendor supplied release notes, a third-party audit report, and one or more symbol maps in `/app`. Treat them as untrusted clues; the ground truth is in the firmware binaries. The day-N+1 image contains several documented changes; exactly one of them is a malicious backdoor. Produce a forensic report that identifies the backdoor. Write these four artifacts under `/logs/artifacts/`: 1. `offset.txt` — the file offset (hex, e.g. `0x1a4`) of the patched instruction in `firmware_v2.bin`. 2. `function.txt` — the name of the bootloader function where the patch was inserted. 3. `trigger.txt` — the certificate extension OID or field value that activates the backdoor (hex, e.g. `0x00badc0d`). 4. `payload.txt` — the exact ASCII payload/secret that the patched code exfiltrates. All hex values should be lower-case; `offset.txt` may include or omit leading zeros.

  4. agent

    I’m going to diff the two firmware images, inspect the bootloader symbols and any vendor-provided notes, then trace which code change is actually behaviorally suspicious in the binaries.

  5. agent

    Executed exec_command call_ws1qeQOTN5oBeKOGgwvuTdFD

  6. agent

    Executed exec_command call_9x0BdoFdyA7N3lyMjKMglcgY

  7. agent

    I have the candidate inputs. Next I’m locating the exact byte-level differences between `firmware_v1.bin` and `firmware_v2.bin`, then I’ll map those offsets back to bootloader functions.

  8. agent

    Executed exec_command call_0hjLQmM4geXKZlQS3U5gqeS6

  9. agent

    Executed exec_command call_oAoSkitqBXmLngcX7kgajGJq

  10. agent

    Executed exec_command call_GOHqt1SGE10MWP64PrhFK05E

  11. agent

    The notes and audit already point at a likely story, but I’m treating them as hints only. I’m now disassembling the bootloader and reconciling the actual changed bytes against symbol addresses.

  12. agent

    Executed exec_command call_Hky6WcjmQFIvH1XsfhCdQCJg

  13. agent

    Executed exec_command call_wPvH3R4lVIe9jwIPs4wbrMlz

  14. agent

    Executed exec_command call_rLLaseUplWR7ybgjCO5IwwYQ

  15. agent

    The provided `map.json` has one candidate patch at `0x70` inside `cert_validate`, and the decoy map contradicts it. I’m going to verify directly from the ELF and raw image bytes before writing artifacts.

  16. agent

    Executed update_plan call_AQT7SfujxRuvLGmj1n1XEXTC

  17. agent

    Executed exec_command call_Y2xmCEdJpWzKEB7aBVjN2Tek

  18. agent

    Executed exec_command call_IYafcQAkb8YeFCDJG9lnXHS4

  19. agent

    Executed exec_command call_Zd6GzgGL2dhNFSoXzp2AFBCd

  20. agent

    Executed exec_command call_sVkXFffX01F1ovqs0e8e3m10

  21. agent

    Executed exec_command call_dtvdtB7KfvlgXS3vDLwCrwO1

  22. agent

    Executed exec_command call_WHB3KIL7iUH3G1QXw60kmGoE

  23. agent

    The code diff is narrow: five changed instructions/operands, but only one introduces a new secret-copy path. I’m decoding the v1 branch targets directly so the report rests on the binaries, not the vendor map.

  24. agent

    Executed exec_command call_6Vm3es6KNHCFg1HfBFb2XM0l

  25. agent

    Executed exec_command call_Cj869jtAIL5ki8r0sCtsajNR

  26. agent

    Executed exec_command call_nWvOJTrAWyZZJznZrTWM1aTS

  27. agent

    I’ve confirmed the only control-flow patch that unlocks new behavior is the branch at `0x70` in `cert_validate`; the `hash_update` change is structurally similar but consistent with a documented policy path. I’m extracting the exact ASCII value that flows out of the legacy path now.

  28. agent

    Executed exec_command call_qnLCYySXZzfnqwOkzPLiJTlz

  29. agent

    Executed exec_command call_5B0NdlRp1D7KzRTlOyXucAHZ

  30. agent

    Executed exec_command call_bTc0Ljj2tQqaIrc1IwrhbDjp

  31. agent

    Executed exec_command call_019RbjNNgbnGbHMXRNEQAEgC

  32. agent

    Executed exec_command call_WCbT9mTj4EccRAHDl6hrLcgj

  33. agent

    Executed exec_command call_SoI5AY173XnZRR4G27Z31qDd

  34. agent

    I have the ground-truth values from the binaries. I’m writing the four artifact files now and then I’ll do a quick read-back to confirm the contents.

  35. agent

    Executed apply_patch call_PYNWzRSvcflY7Zzt5R8drMxO

  36. agent

    Executed exec_command call_vAt2QhcItOTnezBOagSr6YWs

  37. agent

    Executed exec_command call_0ZtSsxgjRgqt6DK1WR47hRfR

  38. agent

    Executed update_plan call_46qLEO8pZ71Z3TzfYcsw6ysm

  39. agent

    The artifacts are written under [/logs/artifacts](/logs/artifacts): `cert_validate` contains the malicious patch. The patched instruction starts at offset `0x70` in `firmware_v2.bin`, accepts the legacy certificate OID `0x00badc0d`, and unlocks a path that yields the payload `exfiltrated_core_secret=RV32I_PATCH_BACKDOOR`. Files: - [/logs/artifacts/offset.txt](/logs/artifacts/offset.txt) - [/logs/artifacts/function.txt](/logs/artifacts/function.txt) - [/logs/artifacts/trigger.txt](/logs/artifacts/trigger.txt) - [/logs/artifacts/payload.txt](/logs/artifacts/payload.txt)