Skip to content

Move the Linux CUDA wheel workflow to OSDC runners - #1849

Merged
NicolasHug merged 8 commits into
mainfrom
osdc-linux-cuda-wheel
Nov 20, 2026
Merged

NicolasHug merged 8 commits into
mainfrom
osdc-linux-cuda-wheel

Conversation

@huydhn

@huydhn huydhn commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Move linux_cuda_wheel.yaml from EC2 to OSDC runners.

  • build: runner-fleet: osdc. PRs run on mt-l-x86aavx2-11-41-a10g. Release refs (main, nightly, release/*, v* tags) use the same regular runner: meta-pytorch can't use the release runners (Keep non-pytorch orgs on regular OSDC runners on release refs pytorch/test-infra#9485).
  • install-and-test, build-docs: linux.g5.4xlarge.nvidia.gpu → mt-l-x86aavx2-11-41-a10g. Docs are still built on every PR from the wheel this workflow builds. upload-docs (main and release tags) is unchanged: it pushes to gh-pages with GITHUB_TOKEN and needs no AWS credentials.
  • Docs preview: OSDC pods have no AWS credentials, so the S3 upload needs GitHub OIDC. A fork's pull_request run can't get an OIDC token, so doc-preview moves to a new docs_preview.yaml. It's triggered by workflow_run when Linux CUDA x86 completes, and runs from main in this repo. It finds the PR from the run's head, downloads Built-Docs from that run, and uploads it with role/arc on mt-l-x86iavx512-8-64. This covers fork and same-repo PRs, and needs no label. workflow_run only uses the copy of the workflow on main, so it starts working after this lands.

@pytorch-bot

pytorch-bot Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1849

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 378abdf with merge base 0fc8b84 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 12, 2026
@huydhn
huydhn force-pushed the osdc-linux-cuda-wheel branch from bf4e523 to 0964461 Compare November 16, 2026 23:39
@huydhn
huydhn requested review from NicolasHug and atalman November 17, 2026 00:17
@huydhn
huydhn marked this pull request as ready for review November 17, 2026 00:17
Comment thread .github/workflows/linux_cuda_wheel.yaml Resolved
@huydhn
huydhn force-pushed the osdc-linux-cuda-wheel branch from 0964461 to e9903e8 Compare November 17, 2026 05:20
Comment thread .github/workflows/linux_cuda_wheel.yaml Outdated
Comment on lines +400 to +401
if: github.repository == 'meta-pytorch/torchcodec' && github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.fork == false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a regression, we rely on this doc PR preview job often and we always submit fork PRs, since it's the normal GitHub workflow.

I could work around this by submitting non-fork PRs, although I'd prefer not to, but external contributors are unable to do so anyway.

Is there a planned fix for this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. There is an extra step we need to do to maintain the existing behaviour, but it's doable. Basically, the doc build needs to be trigger via ciflow so that it runs in the context of meta-pytorch/torchcodec instead of the fork. Then the job will have the permission to upload the preview to S3. Let me update the PR for this

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in f491103. Fork PRs can get a preview again: a maintainer adds the ciflow/docs label, and pytorch-bot pushes a ciflow/docs/<PR> tag. That run happens in meta-pytorch/torchcodec, so it can assume the role and upload to the same meta-pytorch/torchcodec/<PR> preview path. It builds the wheel and docs and skips the GPU tests. Same-repo PRs still upload directly. The bot reads .github/pytorch-probot.yml from main, so the label only works after this PR lands, and the ciflow/docs label needs to be created.

@huydhn huydhn Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decide to move doc build logic to a new workflow to make it easier to trigger it instead of keeping it in the CUDA workflow.

Comment thread .github/workflows/linux_cuda_wheel.yaml Resolved
Comment thread .github/workflows/linux_cuda_wheel.yaml Resolved
@huydhn

huydhn commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@huydhn
huydhn requested a review from NicolasHug November 18, 2026 05:44
@NicolasHug

Copy link
Copy Markdown
Contributor

Thanks for handling the PR preview. I'd like to go back to the f491103 approach. Docs need to be built on every PR, not only when the label is there: building them is our only check that docs and tutorials don't break, and we want to catch that before the merge.

Concretely:

  • Keep build-docs in linux_cuda_wheel.yaml, running on every PR and reusing the wheel that's already built.
  • Keep upload-docs there too, for main and release tags.
  • The ciflow/docs label should only gate the fork-PR preview upload, since that's the only part that needs the
    OIDC role. Same-repo PRs should still upload their preview directly, as in f491103.
  • On ciflow/docs runs, only build the cu132 / py3.11 wheel the docs need, and skip all the test jobs (not just
    install-and-test).
  • Please add examples/** to the labeler rule, since that's where the tutorials live.

@huydhn

huydhn commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Let me try to explain a bit better on why we need a work around here to address your points. The key change is that OSDC runners as k8s pods don't have any credential to write anything. Previously, bare metal EC2 runners have a blanket default permission to write anything anywhere they like on the artifact bucket, which is very bad for security reason. Because OSDC runners don't have any credentials now, they need to authenticate to get one before they can write. This applies to all cases, fork, non-fork, main, release. Now, the GitHub OIDC authentication works roughly like this:

  • If the branch / ref is from meta-pytorch/torchcodec, it's trusted and GitHub gives the credentials. This is why a non-fork PR can authenticate, and also why ciflow works as a alternative because PyTorch bot itself pushes ciflow/docs/PR_NUMBER ref to meta-pytorch/torchcodec. Basically, this is authenticated using the bot identity.
  • If the branch is from a fork, i.e. huydhn:my-branch, GitHub flats out reject the request. You can't trust an "unknown" fork. This means that triggering an upload job from a fork branch will never work. There is no bypassing this.

Another thing is about GitHub workflow triggers, you can have multiple triggers in the workflow, i.e pull_request, ciflow, but they run as different workflow runs. This means that moving the doc jobs to another workflow is to avoid running all the CUDA wheel jobs twice in two identical runs of the same workflow.

Now, let me go over your points one by one:

  • ✅ Keep build-docs in linux_cuda_wheel.yaml, running on every PR and reusing the wheel that's already built <-- This is doable
  • ⚠️ Keep upload-docs there too, for main and release tags <-- The docs-preview job needs to go to a different workflow, it can't stay and expect to work for all cases. However, upload-docs can stay as it's run only on main and release anyway.
  • ❌ The ciflow/docs label should only gate the fork-PR preview upload, since that's the only part that needs the OIDC role. Same-repo PRs should still upload their preview directly, as in f491103<-- Uploading directly can't be done as there is no credential now without OIDC authentication.
  • ✅ On ciflow/docs runs, only build the cu132 / py3.11 wheel the docs need, and skip all the test jobs (not just install-and-test) <-- No concern here.
  • ✅ Please add examples/** to the labeler rule, since that's where the tutorials live <-- No concern here too

In conclusion, I think we need to move docs-preview. This is not as standard as using ciflow IMO as CUDA test and docs build seems like 2 separate things. Nevertheless, this is your call and it's doable using GitHub as an intermediate, so the flow becomes:

  1. linux_cuda_wheel.yml runs first, it uploads the docs to GitHub artifacts instead of S3. This works for all use cases including forks
  2. Create a new upload workflow to download the artifacts from the previous step and upload to S3. This is trigged by a workflow_run owned by the repo and not a pull_request. So, it can authenticate. Here is an example FYI https://github.com/pytorch/pytorch/blob/main/.github/workflows/upload-test-stats.yml#L5

Note that this new approach has a drawback that docs-preview only run after the whole linux_cuda_wheel.yaml finishes (the trigger). That's why this isn't done on PyTorch as most folks don't want to wait for pull to finish before doc preview becomes available. That concern probably doesn't apply here as you want docs to reuse the wheel from the CUDA jobs, so it needs to wait anyway.

Keep build-docs and upload-docs in linux_cuda_wheel so docs are built on
every PR. Preview uploads need OIDC, which fork pull_request runs can't
get, so a workflow_run workflow uploads the Built-Docs artifact instead.
This replaces the ciflow/docs workflow and labeler.
@huydhn
huydhn force-pushed the osdc-linux-cuda-wheel branch from bf7b977 to 87f6dd2 Compare November 18, 2026 18:42
meta-codesync Bot pushed a commit to meta-pytorch/torchrec that referenced this pull request Nov 19, 2026
Summary:
Move the doc preview upload off EC2 (`linux.2xlarge`) so it works for both fork and same-repo PRs.

- OSDC pods have no AWS credentials, so the S3 upload needs `role/arc` through GitHub OIDC. A fork's `pull_request` run can't get an OIDC token.
- `doc-preview` moves from `docs.yml` to a new `docs_preview.yml`, triggered by `workflow_run` when Generate documentation completes successfully. It runs from `main` in this repo, so it can authenticate for any PR. It finds the PR from the run's head, downloads `Built-Docs` from that run and uploads it to `doc-previews/meta-pytorch/torchrec/<PR>` on `mt-l-x86iavx512-8-64`.
- `build_docs_job` and the `gh-pages` deploy are unchanged.
- `workflow_run` only uses the copy of the workflow on `main`, so this starts working after it lands. The same workflow is in meta-pytorch/torchcodec#1849, where it was tested on a fork and a same-repo PR.

Pull Request resolved: #5092

Reviewed By: aporialiao

Differential Revision: D133333925

Pulled By: huydhn

fbshipit-source-id: 32f256c265ea8ddcce63baa40123c31d0d07b775

@NicolasHug NicolasHug left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the extra work making sure we preserve the preview! IIUC we can only try it out on the next PR, so I'll merge this and trigger a test PR. Thanks!

@NicolasHug
NicolasHug merged commit 4521a43 into main Nov 20, 2026
103 of 104 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants