Join worker vCPU threads before tearing down guest memory

[Max042004]

On exit_group the main thread left vcpu_run_loop and unmapped the guest slab
via guest_destroy() while sibling vCPU threads could still be mid-iteration
touching guest memory: the host process died with SIGSEGV and a guest
exit_group(0) was reported as exit 139. Masked until the fault-delivery fix
let multi-threaded JVM workloads (javac) reach the exit_group teardown.

• main() joins workers after vcpu_run_loop() returns and before any teardown;
  gdb_stub_shutdown() runs first so a worker parked in a GDB stop can
  deactivate instead of timing out the join.
• sc_exit_group no longer joins: it ran on the requesting thread, snapshotted
  never-deactivating slot 0, burned the poll cap, detached the host main
  pthread, and raced main()'s join into a double pthread_join. It now only
  requests + interrupts; main() holds the single authoritative join.
• Timed futex waits slept to the guest deadline, unreachable by teardown for
  the wait's whole duration (the FUTEX_WAIT_BITSET condvar path that glibc
  timed waits and JVM parkNanos issue never routes to os_sync; same in
  futex_lock_pi and futex_waitv). They now sleep in 100 ms quanta and re-check
  exit_group/futex_interrupt each quantum; guest-visible timeout accuracy is
  unchanged.
• thread_join_workers() polls under one shared 500 ms deadline — the old
  100 ms per-worker cap sat below the io wait's 200 ms flag re-check bound —
  and marks timed-out workers join_abandoned so guest_destroy()'s later pass
  cannot join/detach the same pthread twice (UB for a detached thread).
• Every thread_entry_t.active write is now a release store: the joiner polls
  lock-free, so the store/load pair on active is its only happens-before edge
  (flagged by ThreadSanitizer on the teardown test; clean under TSan with the
  fix).
• test-exit-group-worker drives a worker-issued exit_group with siblings
  pinned in each blocking state the join relies on (memory spinners, a timed
  futex wait 600 s out, a blocking pipe read) through 10 fork-child teardowns
  plus the top-level path.

A correctness argument — teardown paths, per-state wake bounds vs the join
cap, and residual risk — follows in a comment.

---

Summary by cubic

Join worker vCPU threads in main before tearing down guest memory to prevent host SIGSEGVs and keep correct exit codes. Add a pre-join exit/interrupt kick and bound long waits (futex, FUSE, netlink, wait4) so workers always notice teardown; shut down the debugger first to avoid join timeouts.

• Bug Fixes

  • Call gdb_stub_shutdown() before thread_join_workers().
  • Join workers in main after vcpu_run_loop(); if no exit was requested, request exit_group(0) then call futex_interrupt_request(), wakeup_pipe_signal(), and thread_interrupt_all(); sc_exit_group now only requests and interrupts to avoid double-join and main-thread detach.
  • In guest_destroy(), run request/interrupt/join when destroy is reached directly so workers can’t touch freed memory.
  • Futex waits: sleep in 100 ms quanta, re-check exit/interrupt each quantum, and also re-check expired itimers and queued signals; futex_waitv quantum reduced to 100 ms; timeout accuracy unchanged.
  • thread_join_workers() polls under one shared 500 ms cap and marks timed-out workers join_abandoned so later passes skip them.
  • All writes to thread_entry_t.active are release stores so the lock-free join path observes a happens-before edge.
  • Make FUSE daemon reads, netlink waits, and specific-pid wait4 blocking paths bounded and interrupt-aware so teardown can reach them and avoid post-unmap memory access.

• Tests

  • Add test-exit-group-worker (and build target) to verify worker-issued exit_group exits cleanly in forked children and in the top-level path.

^{Written for commit 1fbbebb. Summary will update on new commits.}

[Max042004]

(A) Paths: every teardown path establishes the precondition before unmapping

main() orders shutdown → join → diagnostics → destroy (src/main.c:675-702):

    gdb_stub_shutdown();          /* must precede the join: a worker parked in
                                     gdb_stub_handle_stop() stays active until
                                     resume_cond broadcasts */
    thread_join_workers();
    ...
    if (shim_globals_stats_enabled())
        shim_globals_counters_dump(&g);   /* reads guest memory — must sit
                                             between join and destroy */
    syscall_hist_dump();
    cleanup_main_resources(...);          /* → guest_destroy() */

fork_child_main() has nine pre-vCPU failure exits (no workers can exist yet)
plus the normal exit; it depends on nothing the caller did because
guest_destroy() re-runs the full sequence itself (src/core/guest.c:621-626):

    if (!proc_exit_group_requested())
        proc_request_exit_group(0);
    futex_interrupt_request();
    wakeup_pipe_signal();
    thread_interrupt_all();
    thread_join_workers();

On the main() path this internal join sees an empty snapshot (workers
already reaped, active clear).

(B) the join terminates — every blocking state has a wake-up bounded

below the join cap

Worker state	Wake mechanism	Bound
inside hv_vcpu_run (incl. shim EL1 fast path)	hv_vcpus_exit → CANCELED → flag check	µs
host handler, not blocked	flag check at loop top	handler runtime
poll/select/epoll_pwait	wakeup pipe; 200 ms re-arm re-check	≤ 200 ms
interruptible io wait (pipe/socket/tty rw)	wakeup pipe + 200 ms flag re-check	≤ 200 ms
futex, untimed (condvar) / os_sync	100 ms quanta + flag check	≤ 100 ms
futex, timed (condvar) — FUTEX_WAIT_BITSET, i.e. glibc timed waits, never routed to os_sync	was: slept to guest deadline, unbounded → 100 ms quanta; same fix in futex_lock_pi; futex_waitv 500→100 ms	≤ 100 ms
gdb_stub_handle_stop()	resume_cond broadcast before join	immediate

The 200 ms io bound (src/syscall/io.c:169-178) — the pipe is
single-consumer, a sibling can steal the wake byte, hence the bounded re-check:

    for (;;) {
        if (proc_exit_group_requested() || futex_interrupt_consume() ||
            signal_pending_interruption(NULL))
            return -LINUX_EINTR;
        int ret = poll(fds, 2, 200);
        ...

The unbounded timed-futex state and its fix (futex_wait, identical pattern
in futex_lock_pi at src/runtime/futex.c:1205):

     while (!__atomic_load_n(&waiter.woken, __ATOMIC_ACQUIRE)) {
         if (has_timeout) {
-            int rc = pthread_cond_timedwait(&waiter.cond, &b->lock, &deadline);
-            if (rc != 0) {
-                /* Timeout (ETIMEDOUT) or error; stop waiting */
+            struct timespec quantum;
+            if (!futex_quantum_deadline(&deadline, &quantum)) {
                 ret = -LINUX_ETIMEDOUT;
                 break;
             }
+            pthread_cond_timedwait(&waiter.cond, &b->lock, &quantum);
+            if (proc_exit_group_requested() || futex_interrupt_consume()) {
+                ret = -LINUX_EINTR;
+                break;
+            }
             continue;
         }

futex_quantum_deadline() (src/runtime/futex.c:382) caps each sleep at
FUTEX_OS_SYNC_POLL_CAP_NS (100 ms) or the remaining guest deadline,
whichever is sooner — so guest-visible timeout accuracy is unchanged.

Join cap was 100 ms per worker — below the 200 ms io bound and vacuous against
the unbounded state. Now one shared 500 ms deadline
(src/runtime/thread.c:366-374), 2.5× the worst bound, not multiplied by
worker count; the poll exits on deactivation, so the common case stays µs:

    for (int i = 0; i < 100; i++) {
        bool any_active = false;
        for (int w = 0; w < nworkers; w++)
            if (__atomic_load_n(&workers[w].t->active, __ATOMIC_ACQUIRE)) {
                any_active = true;
                break;
            }
        if (!any_active)
            break;
        usleep(5000);
    }

(C) the join itself is UB-free

Double join/detach: a worker detached by a timed-out pass stayed active, so
guest_destroy()'s second pass would join/detach the same pthread — UB for a
detached thread. Now marked and skipped
(src/runtime/thread.c:343, :379-388):

        if (t == current_thread || t->join_abandoned)   /* snapshot skip */
            continue;
    ...
        if (!__atomic_load_n(&workers[w].t->active, __ATOMIC_ACQUIRE)) {
            pthread_join(workers[w].thr, NULL);
        } else {
            pthread_detach(workers[w].thr);
            pthread_mutex_lock(&thread_lock);
            workers[w].t->join_abandoned = true;
            pthread_mutex_unlock(&thread_lock);
        }

active was a plain store under thread_lock vs the joiner's lock-free
acquire-load — the mutex gives the joiner no HB edge; without release the
compiler may sink the worker's final guest stores past the flag write. Found
by TSan on the teardown test; every write is now a release store
(src/runtime/thread.c:226, in thread_deactivate):

    __atomic_store_n(&t->active, 0, __ATOMIC_RELEASE);

so: every guest access happens-before the active release-store,
happens-before the joiner's acquire-load, happens-before the unmap. vCPUs are
destroyed only after the join (thread_destroy_all_vcpus), never cross-thread
while the owner may be in hv_vcpu_run.

Residual

Host condvar parks outside the syscall wait set (fork_cond mid-fork-barrier,
ptrace_cond mid-ptrace-stop) are not in the table; a worker there can exceed
the cap, be detached, and I then holds only because process exit follows
guest_destroy() almost immediately — mitigation, not proof. Same trade-off
as the pre-existing cross-thread hv_vcpu_destroy for abandoned workers.

Evidence (necessary, not sufficient)

• test-exit-group-worker: worker-issued exit_group with siblings pinned in
  the load-bearing states — 3 spinners, one timed FUTEX_WAIT_BITSET 600 s
  out (sleeps through any cap on the old code), one blocking pipe read —
  through 10 fork-child teardowns + the top-level path. 11/11, ~1.6 s total
  (one uncapped timed-futex sleep would exceed that by 100×). make check
  green.
• TSan (host side only — stage-2 accesses inside hv_vcpu_run are invisible
  to it): flagged the active plain-store/atomic-load pair, fixed above.
  With the fix in and the known race suppressed:
  test-exit-group-worker, test-futex-pi, test-futex-waitv,
  test-robust-futex, test-pthread — zero reports.

[jserv]

Rebase latest main branch, resolve conflicts, and refine per review messages.