Fail signal-killed pipelines instead of ingesting their partial output

[mattdawkins]

Problem

stream_subprocess only treated a positive return code as failure:

code = process.wait(30)
...
if code > 0:
    raise RuntimeError(...)
else:
    manager.write(f"\nProcess completed in ...")

Popen reports death by signal as a negative return code. A pipeline killed by SIGKILL — most commonly the OOM killer, at -9 — fails the code > 0 test and falls into the success branch.

Why it corrupts data

Callers act on that "success". run_pipeline proceeds straight to PUSHING_OUTPUT, uploads the output file the dead pipeline left behind, and posts it to postprocess, which imports it over the dataset's existing annotations.

So a pipeline OOM-killed partway through replaced the user's annotations with a truncated CSV — or an empty one — and the job still reported success.

Reproduced directly:

$ bash -c 'echo partial-output; kill -9 $$'
child stdout     : b'partial-output\n'
Popen returncode : -9
  'code > 0'  -> False   => success branch, partial output ingested
  'code != 0' -> True    => raises, upload/import skipped

Fix

Treat any nonzero code as failure, and name the signal in the error so an OOM-killed run is identifiable from the job log (was terminated by SIGKILL (9) rather than a bare number).

Cancellation is unaffected — it is detected and raised before this check.

Tests

server/tests/test_stream_subprocess.py runs real subprocesses, including one that SIGKILLs itself, and asserts it raises rather than returning normally.

🤖 Generated with Claude Code

[BryonLewis]

Looks good