Skip to content

feat(llm): add Codex app-server provider for ChatGPT subscription auth - #512

Merged
codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
akushonkamen:fix/484-codex-app-server
Oct 13, 2026
Merged

codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
akushonkamen:fix/484-codex-app-server

Conversation

@akushonkamen

Copy link
Copy Markdown
Contributor

Fixes #484

Summary

Add an opt-in codex_app_server LLM provider so Codex / ChatGPT subscription users can run OpenEvolve without an API key (as requested in #484, following @codelion's guidance to go through the Codex app-server rather than the CLI's token files).

  • New provider codex_app_server (openevolve/llm/codex_app_server.py): talks to codex app-server over its stdio JSONL JSON-RPC transport — initialize handshake → account/read (auth boundary) → thread/start → turn/start. The app-server owns and refreshes the OAuth tokens; OpenEvolve never reads the CLI's token files.
  • Auth boundary: before generation, account/read must report a ChatGPT-plan login. Unauthenticated accounts (or accounts running in API-key mode) fail fast with actionable codex login instructions and are not retried — retrying cannot fix an auth error.
  • Non-interactive safety: threads run with approvalPolicy=never and a read-only sandbox, so generation cannot mutate the workspace.
  • Registry wiring: registered as "codex_app_server" in _PROVIDER_REGISTRY (openevolve/llm/ensemble.py) and re-exported from openevolve/llm/__init__.py, following the existing copilot_cli provider pattern.
  • README: new provider section documenting setup (codex login, then provider: codex_app_server), auth modes, and failure modes.

Motivation

OpenEvolve's LLM backends hard-require an API key: LLMModelConfig.api_key (openevolve/config.py:56, env-resolved in __post_init__ at openevolve/config.py:100-101) is consumed by OpenAILLM (openevolve/llm/openai.py:63) and injected into the openai.OpenAI client (openevolve/llm/openai.py:87-88). Users on a Codex subscription have no API key at all, so they could not run custom harnesses against their subscription. As of the issue baseline (HEAD 30d9752) the repo had zero Codex integration: git grep -i codex in openevolve/ → 0 hits.

The default API-key path is untouched: openevolve/config.py and openevolve/llm/openai.py have zero changes in this PR, and a config with no provider still routes to OpenAILLM.

Scope notes

  • openevolve/llm/__init__.py and openevolve/llm/ensemble.py already fail isort --check on main (verified against 30d9752); this PR keeps the files' existing import arrangement rather than reordering unrelated imports.
  • One intermediate full-suite run showed a single failure in tests/test_sample_from_island_ratios.py; reproduced on clean main via git stash — flaky, unrelated to this change.

Testing

New fully-offline suite tests/test_codex_app_server_llm.py (24 tests): an in-process fake app-server driven over real OS pipes, so the client code exercises its actual JSON-RPC machinery end to end — no network, no real codex binary. Covers: auth accept (ChatGPT account) / reject (unauthenticated) / reject (API-key mode, asserting no turn/start is ever sent), protocol ordering, interrupted / crashed / hung servers, retry behavior, registry wiring, and config propagation.

Test evidence (this branch; the system has no pip/pytest, so commands ran via a uv-created .venv interpreter):

$ timeout 300 .venv/bin/python -m pytest tests/ -k codex_app_server -x -q
24 passed, 664 deselected in 2.25s

$ timeout 420 .venv/bin/python -m pytest tests/ -q
686 passed, 2 skipped, 1 warning, 84 subtests passed in 57.06s

The 2 skips are pre-existing scipy skips (tests/test_circle_packing_best_program.py: 2 skipped), and the 1 warning is a pre-existing PytestReturnNotNoneWarning in tests/test_regional_endpoint.py — a file this PR does not touch.

Formatting: black --check --line-length 100 (the repo's pyproject settings) clean on all 4 touched .py files.

Development was red → green: the initial failing tests caught three real bugs before going green — a helper renamed without updating a call site; CodexAuthError wrongly being retried; notifications from the app-server being dropped while a response was awaited.


Transparency note: this contribution is prepared with AI assistance (an automated coding agent), reviewed and submitted by me — same disclosure as in the issue claim comment.

Closes algorithmicsuperintelligence#484

OpenEvolve's LLM backends require an API key, so Codex subscription
users could not run custom harnesses against their subscription. Add an
opt-in `codex_app_server` provider following the maintainer's guidance:

- Talks to `codex app-server` over its stdio JSONL JSON-RPC transport
  (initialize handshake -> thread/start -> turn/start), never reads the
  CLI's token files; tokens are owned and refreshed by the app-server.
- Auth boundary: verifies a ChatGPT-plan login via account/read before
  every generation; missing or API-key-mode accounts fail fast with
  actionable `codex login` instructions and are not retried.
- Threads run non-interactively (approvalPolicy=never, read-only
  sandbox), so generation cannot mutate the workspace.
- The default OpenAI (API-key) path is untouched; no provider configured
  still routes to OpenAILLM.

Tests drive an in-process fake app-server over real OS pipes (no
network, no real binary): auth accept/reject at the boundary, protocol
ordering, interrupted/crashed/hung servers, retries, registry wiring,
and config propagation. Docs: new README provider section.

Signed-off-by: akushonkamen <akushonkamen@163.com>
@codelion
codelion merged commit 15f2b9a into algorithmicsuperintelligence:main Oct 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Using Codex Subscription tokens

2 participants