Skip to content

fix(api): answer 404, not 500, when a request loses a race to a delete - #812

Merged
leoparente merged 1 commit into
developfrom
fix/module-race-404
Nov 16, 2026
Merged

leoparente merged 1 commit into
developfrom
fix/module-race-404

Conversation

@leoparente

Copy link
Copy Markdown
Contributor

The admin API routes that read or remove one module checked that it existed, then looked it up again to act on it. A DELETE landing between the two made the second lookup throw, so the route answered 500 module name '<name>' does not exist instead of the 404 it gives for a module that never existed. The __all metrics routes failed a whole scrape with 500 when one policy was deleted before its turn.

A race the API can lose is a known, repeatable case, so it should never answer 500. A module that is gone answers 404.

Routes

Route Answer when the module is gone
GET /api/v1/taps/{name} 404 tap does not exists
DELETE /api/v1/taps/{name} 404 tap does not exists (since #809; now tested)
GET /api/v1/policies/{name} 404 policy does not exists
DELETE /api/v1/policies/{name} 404 policy does not exists
GET /api/v1/metrics/bucket/{n}, /window/{n} (default policy) 404 no "default" policy exists
default policy Prometheus path 404, empty body
GET /api/v1/policies/{name}/metrics/(window|bucket)/{n} 404 policy does not exist
GET /api/v1/policies/{name}/metrics/prometheus 404 policy does not exists (text/plain)
__all on both per-policy metrics routes 200; a policy removed since it was listed is skipped, as the OpenTelemetry push already does

Each route keeps the message it already used for a missing module. For a module that exists, or one that never existed, every route answers exactly as before: status, body and content type.

The fix

  • AbstractManager::module_find_locked(name) finds and locks a module in one step and returns nullptr when it is missing. The routes use it instead of module_exists followed by module_get_locked, so there's no gap between the check and the lookup, and the 404 no longer depends on exception types.
  • The DELETEs answer 404 to the ModuleException that remove_tap and remove_policy throw, having changed nothing, when no module has the name. Any other failure still answers 500. The contract is documented on both declarations.
  • remove_tap checks for the tap before it waits for a policy load, so a DELETE of a tap that's already missing answers at once instead of waiting behind a load. A load can read a whole pcap file while it holds that lock. A tap that exists still goes through the locked re-check.
  • A single policy's metrics look the policy up before parsing the period, so a missing policy answers 404 whatever the period, as before.
  • The route bodies moved to functions in visor::api (CoreServer.h). The lambdas only parse the request, so tests can call a route in the state a lost race leaves.
  • visor-core links CoreFoundation and Security on macOS. The test binary now links CoreServer.cpp, whose HTTPS client loads keychain roots; pktvisord only got those frameworks through crashpad. Linux and Windows are unchanged.

Tests

  • src/tests/test_core_server.cpp (new). Each case loads a module, removes it, then calls the route function and asserts the status and JSON error. It also checks a 200 before the removal. Covered:
    • the tap routes;
    • the policy routes;
    • the default-policy routes;
    • the per-policy metrics, including a period that overflows long, __all skipping a removed policy, __all skipping a policy without that period yet, and 425 for a single one.
  • src/tests/test_policies.cpp:
    • 100 rounds of 4 concurrent remove_policy calls: exactly one removal per round, and ModuleException for the rest.
    • A DELETE of a missing tap answers 404 within 5 s while a load is held at its first registration, using fix(policies): delete a tap without deadlocking or racing a policy load #809's gated manager.
    • Two DELETEs of a tap that both passed the early check while a load was held: exactly one 200 and one 404 once the load finishes.
    • All of them have a 180 s watchdog.
  • Mutations. Each of these made its test fail:
    • the lookups reverted to the throwing module_get_locked (develop's behaviour);
    • the DELETE catches removed;
    • the __all skip removed;
    • the period parsed before the lookup;
    • the early tap check removed;
    • the locked tap check removed, which segfaults.

Verification

  • ctest on macOS: 24/25. unit-tests-input-netprobe fails the same way on develop on this machine and passes in CI.
  • e2e, 60 s. pktvisord with --admin-api. Racing cycles each create a module, then release a DELETE together with GETs, DELETEs and metrics reads of the same name, while a scraper loops over the three __all routes:
develop (c3c5379) this branch
500 answers 5,786 (every route above, plus 1,028 __all scrapes) 0
pktvisord stayed up stayed up

Behaviour to know

A DELETE of a tap that exists still waits for a policy load in progress, as since #809.

Not changed

POST races are fixed in a separate PR stacked on this one:

  • POST /api/v1/taps and POST /api/v1/policies render their response from pointers a concurrent DELETE can free;
  • PolicyManager::load checks an input's existence and fetches it in two steps.

🤖 Generated with Claude Code

The admin API routes that read or remove one module checked that it
existed, then looked it up again to act on it. A DELETE landing between
the two made the second lookup throw, and the route answered 500 with
"module name '<name>' does not exist". The __all metrics routes listed
every policy first, so one deleted before its turn failed the whole
scrape with 500.

- AbstractManager::module_find_locked checks and gets a module in one
  step, and returns nullptr when it is missing. The tap and policy GETs,
  the default policy's bucket, window and Prometheus routes, and the
  per-policy JSON and Prometheus metrics use it, and answer 404 with the
  message each already gave for a missing module.
- A single policy's metrics look the policy up before parsing the
  period, so a missing policy still answers 404 whatever the period.
- __all skips a policy removed since it was listed, as the OpenTelemetry
  push does.
- The tap and policy DELETEs answer 404 to the ModuleException that
  remove_tap and remove_policy throw, having changed nothing, when no
  module has the name. remove_tap checks for the tap before it waits for
  a policy load in progress, so a tap already missing is answered at
  once. Any other failure still answers 500.
- The route bodies move to functions in visor::api, so tests can call
  them in the state a lost race leaves: the module gone after its name
  was checked or listed.
- visor-core links the macOS frameworks that httplib's HTTPS client in
  CoreServer needs, which pktvisord only got through crashpad.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoparente

Copy link
Copy Markdown
Contributor Author

@codex review

@leoparente

Copy link
Copy Markdown
Contributor Author

/ai-review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T18:54:30.729404Z e99991a Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@nbl-ai-review

nbl-ai-review Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net-positive fix: check-then-use races on module lookup/removal are closed by making the lookup atomic under the manager lock (module_find_locked), route logic is faithfully extracted into visor::api with matching behavior, and the DELETE paths correctly map the race outcome to 404 instead of 500. No security-sensitive surface (authn/authz, input validation, secrets) is affected, and the change is well covered by new concurrency and mutation tests.

No findings.

🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 3 comment(s) · 0 prior AI review(s) · usage: $1.4 · 2763k in (96% cached) · 9.2k out

@nbl-ai-review

nbl-ai-review Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net-positive change: it closes the check-then-act races where a concurrent DELETE caused a 500 instead of a 404, via module_find_locked and reordered checks (early tap existence, policy lookup before period parsing), backed by new targeted concurrency tests. No correctness or security issues found.

🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 3 comment(s) · 0 prior AI review(s) · usage: $1.19 · 2468k in (96% cached) · 9.1k out

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: e99991ab84

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

Copy link
Copy Markdown

LCOV of commit e99991a during Debug Builds #284

Summary coverage rate:
  source files: 149
  lines.......: 87.6% (19670 of 22461 lines)
  functions...: 80.5% (2061 of 2560 functions)
  branches....: no data found
Message summary:
  no messages were reported

Files changed coverage rate: n/a

Full coverage report

@leoparente
leoparente marked this pull request as ready for review November 16, 2026 19:44
@leoparente
leoparente merged commit 02acfdf into develop Nov 16, 2026
23 checks passed
@leoparente
leoparente deleted the fix/module-race-404 branch November 16, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants