Sync uv env on head node before sbatch, verify offline on compute node

[tabedzki]

🤖 AI text below 🤖

Summary

• Compute nodes on spock have no network access, so uv sync for the ephys processing repo (BrainCogsEphysSorters) must happen on the head/login node before sbatch submits the job.
• Adds prefetch_uv_env() in slurm_creator.py, called from recording_process_handler.py right before generate_slurm_file/queue_slurm_file: activates the U19-pipeline_python_env3 conda env, cds to the processing repo, and runs plain uv sync (no --frozen/--upgrade — it's allowed to update the env/lockfile to match pyproject.toml, but won't force unrelated upgrades). Runs via the same SSH-vs-local branching (is_this_spock()) already used by queue_slurm_file. On failure, the recording process is marked ERROR_STATUS and returns before sbatch is ever invoked.
• Adds a uv sync --frozen --offline verification step to generate_slurm_spock() (the SLURM template actually used for spock jobs — confirmed generate_slurm_spockmk2_ephys is dead code, never called), right after conda activate and before the job's python call. Since compute nodes have no network, this only succeeds if the head-node sync already matched; on any mismatch it exits non-zero immediately so the job shows FAILED rather than silently running with a stale/incomplete environment.
• conda activate is deliberately kept on both the head node and compute node (not replaced by uv) since conda can install GPU/binary dependencies uv/pip can't provide.

Test plan

• python3 -m py_compile on both changed files
• Verified generate_slurm_spock() emits uv sync --frozen --offline between conda activate and python -u ${process_script_path}
• Verified prefetch_uv_env builds the expected ssh ... command and the local bash -c ... fallback when already on spock
• End-to-end dry run on spock: submit a small ephys job and confirm the head-node prefetch runs before sbatch, and the compute-node log shows no mismatch
• Negative test: desync the environment (e.g. change pyproject.toml without a head-node sync) and confirm the SLURM job fails fast with the mismatch message in its log

🤖 Generated with Claude Code