Skip to content

🤖 bench: target GPT-5.6 Sol and run Terminal-Bench lanes at high thinking#3752

Merged
ibetitsmike merged 1 commit into
mainfrom
mike/tbench-gpt-5-6-sol
Jul 25, 2026
Merged

🤖 bench: target GPT-5.6 Sol and run Terminal-Bench lanes at high thinking#3752
ibetitsmike merged 1 commit into
mainfrom
mike/tbench-gpt-5-6-sol

Conversation

@ibetitsmike

Copy link
Copy Markdown
Contributor

Summary

Updates the Terminal-Bench model targets: the nightly GPT lane moves from GPT-5.5 to GPT-5.6 Sol, and all benchmark lanes now run at --thinking high (Claude Opus 5 previously ran xhigh).

Background

GPT-5.6 Sol replaced GPT-5.5 as the flagship OpenAI tier, and the requested bench configuration is GPT-5.6 Sol at high thinking and Claude Opus 5 at high thinking. The old per-model xhigh/high conditional in the nightly workflow is no longer needed, so it is flattened to a single --thinking high.

Implementation

  • nightly-terminal-bench.yml: default all matrix swaps openai/gpt-5.5 for openai/gpt-5.6-sol; mux_run_args is now a plain --thinking high for every lane. Gemini lanes and the Sonnet smoke tests are unchanged.
  • terminal-bench.yml: model example strings refreshed.
  • prepare_leaderboard_submission.py: adds leaderboard metadata for anthropic/claude-opus-5 (previously missing, so submission prep could not map the current Opus target) and openai/gpt-5.6-sol; older entries stay as historical mappings.
  • tbench skill (and its benchmarks/terminal_bench/README.md symlink): example commands updated to the new targets.

Validation

  • make static-check green locally.
  • YAML parse of both workflows and py_compile of the leaderboard script pass.
  • Grep confirms the only remaining gpt-5.5 references are intentional historical metadata entries.

Generated with mux • Model: anthropic:claude-fable-5 • Thinking: xhigh • Cost: $5.53

@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 6cf31f7c99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike added this pull request to the merge queue Jul 24, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 24, 2026
@ibetitsmike
ibetitsmike added this pull request to the merge queue Jul 25, 2026
Merged via the queue into main with commit 68513bc Jul 25, 2026
21 checks passed
@ibetitsmike
ibetitsmike deleted the mike/tbench-gpt-5-6-sol branch July 25, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant