Skip to content

fix(api): never render a POST response from a module a concurrent delete can free - #813

Merged
leoparente merged 1 commit into
developfrom
fix/post-delete-race
Nov 16, 2026
Merged

leoparente merged 1 commit into
developfrom
fix/post-delete-race

Conversation

@leoparente

@leoparente leoparente commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Follows #812, which this builds on (module_find_locked, the visor::api route functions and their tests).

pktvisord crashed when a DELETE raced the POST that created the same tap or policy. POST /api/v1/taps and POST /api/v1/policies built their 201 response from the raw pointers load_from_str returned, after every manager lock was released. A DELETE landing in between freed the module, and the response read freed memory.

With pktvisord running with --admin-api and a client sending POST and DELETE for the same names at once, develop crashed in 5 runs out of 5, with EXC_BAD_ACCESS in Policy::info_json or in the POST /taps route.

Two more races sat in the same place:

  • The policy load's input lookup. PolicyManager::load checked that an input existed, then fetched it, holding only _load_mutex, which remove_policy didn't take. A DELETE of the last policy on that input in between made the POST roll back and answer 500 module name '<input>' does not exist.
  • An input freed under a new policy. remove_policy saw an input with no policies left and removed it. A load could attach a new policy to that input in between, before the new policy's handlers started, and the input was freed under it.

The fix

  • Each POST answers with JSON rendered inside the load. TapManager::load and PolicyManager::load (and their load_from_str) take an optional json *info, and the routes answer 201 from it, never from the returned pointers.
    • A tap is rendered before it is registered, while only the load holds it.
    • A policy is rendered after it starts, still under _load_mutex, just before it is registered. A render that throws goes through the load's existing rollback, where before it answered 500 for a policy that had already been created.
  • remove_policy takes _load_mutex first, as remove_tap does since fix(policies): delete a tap without deadlocking or racing a policy load #809. No input can be removed while a load is in progress, so the input lookup is atomic with its reuse, and an input can't be freed under a policy that's attaching to it.
  • A DELETE of a missing policy answers 404 without waiting for a load, as fix(api): answer 404, not 500, when a request loses a race to a delete #812 does for taps. A policy that exists is looked up again under the locks before anything is removed.
  • The POST route bodies move to api::post_taps and api::post_policies, like the other routes in fix(api): answer 404, not 500, when a request loses a race to a delete #812, so tests can call them directly.

Lock order: unchanged: _load_mutex < policy map < tap map < input map < handler map. Rendering a policy under _load_mutex takes no manager map lock. _load_mutex is taken only by the load, remove_policy and remove_tap, which never call each other, and the load's rollback doesn't take it.

What a racing client sees

A POST that creates its module always answers 201 with the module as loaded. A DELETE ordered after it answers 200 and removes it; one ordered before it answers 404. Every outcome matches some sequential order of the two requests.

Tests

In src/tests/test_policies.cpp, each with a 180 s watchdog:

  • POST and DELETE of one tap, then of one policy, raced for 5 s. Only 201 or 409 is allowed for POSTs and 200 or 404 for DELETEs, and every 201 body must be complete.
  • POST and DELETE of policies sharing one input, raced for 5 s.
  • A tap POST answers with the tap as loaded when it is removed before the POST answers (deterministic). The load is held just after it registers the tap, the tap is removed, then the load is released. The POST must answer 201 with the tap as loaded, rendered once, before registration.
  • Deleting a policy waits for the POST loading it (deterministic). The DELETE waits, the POST answers 201 with the policy's taps, input and modules, then the DELETE answers 200.
  • A policy POST reusing an input outlasts the DELETE of that input's last policy (deterministic). The load is held between finding the input and taking it, using a debug log line, which the code marks as a test seam.
  • A DELETE of a missing policy answers 404 without waiting for a load.
  • Two DELETEs of a policy that both found it: one removes it, the other answers 404.

In src/tests/test_core_server.cpp, the POST routes answer 201, 409 for a duplicate name, and 422 for a tap selector that matches nothing.

Before the fix: on develop plus only the route extraction, these tests crash (SIGSEGV, or SIGABRT with "Pure virtual function called!" or "mutex lock failed") or fail. The input-reuse test answers 500 every time.

Mutations, each caught by a test:

  • no _load_mutex in remove_policy;
  • no fast path;
  • no re-check under the locks;
  • either POST rendering from the returned pointers;
  • a tap rendered after registration;
  • a policy rendered after _load_mutex is released.

Verification

  • ctest on macOS: 24/25. unit-tests-input-netprobe fails the same way on develop on this machine, and passes in CI.

  • e2e: pktvisord with --admin-api, with clients sending POST, DELETE and GET of the same names at once (10 s chaos runs, a 20 s race run):

    develop this branch
    chaos, each policy with its own input crashed (SIGSEGV), 3 runs out of 3 no crash, no 500, about 8k POST /policies and 9k POST /taps answered 201 per run
    chaos, policies sharing one input crashed (SIGSEGV), 2 runs out of 2 no crash, no 500

Behaviour to know

  • A DELETE of an existing policy now waits for a policy load in progress. That includes a load that reads a whole capture file as it starts. fix(policies): delete a tap without deadlocking or racing a policy load #809 did the same for taps. It can't deadlock: the waiting request holds no lock.
  • The returned pointers are no longer used. load and load_from_str still return raw pointers, documented as freeable once the load ends, but no caller reads them any more.

🤖 Generated with Claude Code

@leoparente

Copy link
Copy Markdown
Contributor Author

@codex review

@leoparente

Copy link
Copy Markdown
Contributor Author

/ai-review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:24:45.411485Z c5cc07f Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@nbl-ai-review

nbl-ai-review Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — This is a net-positive fix: it closes a real use-after-free (CWE-416) where a concurrent DELETE could free a tap/policy module before its POST response was rendered, plus two related races in policy load/remove ordering. Both the correctness and security reviews confirm the fix is applied consistently (matching lock order across TapManager/PolicyManager, responses rendered while still under the relevant lock) and is backed by concurrency tests. No issues found.

🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 3 comment(s) · 0 prior AI review(s) · usage: $1.1 · 1632k in (92% cached) · 10.7k out

@nbl-ai-review

nbl-ai-review Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — This is a net-positive fix: POST responses for taps/policies are now rendered under the load's lock before the module can be freed by a racing DELETE, and remove_policy takes _load_mutex first (matching the existing remove_tap pattern), closing the use-after-free. Lock ordering stays consistent with the documented order, and the new concurrency tests exercise the race windows described. No findings from either dimension.

🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 3 comment(s) · 0 prior AI review(s) · usage: $1.2 · 1973k in (94% cached) · 10.3k out

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: c5cc07f29d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Base automatically changed from fix/module-race-404 to develop November 16, 2026 20:03
…ete can free

POST /api/v1/taps and POST /api/v1/policies built their 201 answer from the
module pointers the load returned, after every manager lock was released. A
DELETE of the same name landing in between freed the module, and info_json
read freed memory: pktvisord crashed.

- Both loads take an optional json to render each module's info_json into
  before it is registered, and both POST routes answer with that. A tap is
  rendered while it is still only the load's own. A policy is rendered once
  it has started, while the load still holds _load_mutex, which every tap
  and policy removal now takes. A POST that creates its module so always
  answers 201 with it as loaded, and a DELETE ordered after it removes it,
  as if it came after the answer.
- remove_policy takes _load_mutex first, as remove_tap does. A load checked
  that the input it reuses existed and took it later, and attached to it
  before its handlers consumed it: a removal of that input's last policy in
  between made the load roll back and the POST answer 500 "module name
  '<input>' does not exist", or freed the input under the new policy, or
  failed the DELETE with 500 and left the policy registered. A policy
  DELETE now waits for a load in progress, as a tap DELETE already does,
  except for a policy already missing, which is reported at once, as
  remove_tap does. The rollback never calls remove_policy, so it cannot
  deadlock on the plain mutex.
- The POST route bodies move to visor::api, like the other routes.
- Tests race POST and DELETE of one tap and of one policy, hold a tap
  load and a policy load just after they register and a policy load at
  its input lookup to order a removal against them, and churn two
  policies on one input. Each crashes or fails without the fix.
  GatedPolicyManager can now hold a load just after a given registration,
  and hold_load_at_gate runs any load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoparente
leoparente marked this pull request as ready for review November 16, 2026 20:22
@leoparente leoparente self-assigned this Nov 16, 2026
@github-actions

Copy link
Copy Markdown

LCOV of commit 978ae6c during Debug Builds #287

Summary coverage rate:
  source files: 149
  lines.......: 87.8% (20144 of 22934 lines)
  functions...: 80.9% (2107 of 2604 functions)
  branches....: no data found
Message summary:
  no messages were reported

Files changed coverage rate: n/a

Full coverage report

@leoparente
leoparente merged commit 446ee82 into develop Nov 16, 2026
23 checks passed
@leoparente
leoparente deleted the fix/post-delete-race branch November 16, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants