fix(vllm): pin L4T arm64 backend to vllm==0.24.0 for GB10 stability#10725
Merged
Conversation
The nvidia-l4t-cuda-13-arm64 vLLM backend left `vllm` unpinned, so the prebuilt image drifted onto whatever aarch64 wheel was latest at build time (0.23.x). On GB10 / DGX Spark (Grace Blackwell, unified memory), 0.23 crashes deterministically during cold model loads with an empty "Engine core initialization failed" set and pins GPU memory until a host reboot. vLLM 0.24.0 carries vllm-project/vllm#45179 ("release cached device memory under pressure on UMA GPUs during weight loading"), which the reporter verified fixes the crash on GB10. Pin the L4T requirements to 0.24.0 to match the already-pinned cublas13 build (requirements-cublas13-after.txt) and keep the image deterministic. Editing this file also re-triggers the single-arch L4T image build via the path filter, republishing the gallery image with 0.24.0 (the single-arch matrix builds again after #10703). Closes #10722 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
mudler
approved these changes
Jul 7, 2026
treilhes
pushed a commit
to treilhes/LocalAI
that referenced
this pull request
Jul 7, 2026
…udler#10725) The nvidia-l4t-cuda-13-arm64 vLLM backend left `vllm` unpinned, so the prebuilt image drifted onto whatever aarch64 wheel was latest at build time (0.23.x). On GB10 / DGX Spark (Grace Blackwell, unified memory), 0.23 crashes deterministically during cold model loads with an empty "Engine core initialization failed" set and pins GPU memory until a host reboot. vLLM 0.24.0 carries vllm-project/vllm#45179 ("release cached device memory under pressure on UMA GPUs during weight loading"), which the reporter verified fixes the crash on GB10. Pin the L4T requirements to 0.24.0 to match the already-pinned cublas13 build (requirements-cublas13-after.txt) and keep the image deterministic. Editing this file also re-triggers the single-arch L4T image build via the path filter, republishing the gallery image with 0.24.0 (the single-arch matrix builds again after mudler#10703). Closes mudler#10722 Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Signed-off-by: treil <p.treilhes@free.fr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Pins the
nvidia-l4t-cuda-13-arm64vLLM backend requirements from unpinnedvllmtovllm==0.24.0.Why
The L4T arm64 vLLM backend left
vllmunpinned inrequirements-l4t13-after.txt, so the prebuilt gallery image drifted onto whatever aarch64 wheel was latest at build time (0.23.x). On GB10 / DGX Spark (Grace Blackwell, unified memory), vLLM 0.23 crashes deterministically during cold model loads with an emptyEngine core initialization failed/Failed core proc(s): {}set and pins GPU memory until a host reboot.vLLM 0.24.0 carries vllm-project/vllm#45179 ("release cached device memory under pressure on UMA GPUs during weight loading"), which the reporter verified fixes the crash on GB10. This also brings the L4T build to parity with the already-pinned
requirements-cublas13-after.txt(x86 cuda13), and keeps the image deterministic instead of drifting on unpinned wheels.Why now (rebuild path)
Editing a file under
backend/python/vllm/re-triggers the vLLM variants via the CI path filter, including the single-arch L4T image build. The single-arch backend matrix only started building again after #10703 (256-job matrix shard fix, merged 2026-07-06) — the three prior weekly crons had silently dropped all single-arch jobs, which is why the L4T image had been frozen at 0.23. With #10703 in master, this change actually republishes the gallery image with 0.24.0.Closes #10722
🤖 Generated with Claude Code