Skip to content

Add agent skill for converting Hugging Face models to TE recipes - #3904

Open
faradawn wants to merge 4 commits into
NVIDIA:mainfrom
faradawn:add-hf-conversion-skill
Open

faradawn wants to merge 4 commits into
NVIDIA:mainfrom
faradawn:add-hf-conversion-skill

Conversation

@faradawn

@faradawn faradawn commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Skill: fusions like LayerNorm+QKV, LayerNorm+MLP, TransformerLayer and GroupedLinear MoE experts, with the checkpoint weight mapping.
Eval: Llama, Gemma and Mixtral

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 19, 2026
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
@faradawn
faradawn marked this pull request as ready for review November 19, 2026 23:20
@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 3/5

[Low impact] The changes affect only users who choose this new recipe-writing skill or run its checker.

The PR needs changes before merging because MXFP8 expert splits remain unpadded and Mistral's attention window remains unpreserved.

Findings

  1. P1 MXFP8 experts need padded tokens. ▶
  2. P1 Mistral loses its attention window. ▶
  3. P2 Ignored weights still fail ▶

Summary

Adds a skill for converting Hugging Face models to fused Transformer Engine recipes.

  • The new skill guides Hugging Face models into Transformer Engine recipes.
  • The checker traces checkpoint weights into a converted Transformer Engine model.

Reviews (4) · Last reviewed commit: "add weight mapping check script to skill" · Reviewed by Greptile

Comment on lines +199 to +203
split_sizes = torch.bincount(experts.reshape(-1), minlength=num_experts).int()
permuted, row_id_map = te.moe_permute( # sort tokens by expert
tokens, experts.to(torch.int32), num_out_tokens=experts.numel(), map_type="index"
)
expert_out = self.experts_ffn(permuted, split_sizes, split_sizes) # one grouped GEMM each

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 MXFP8 experts need padded tokens. The example passes raw expert counts from torch.bincount to GroupedLinear, but the precision hook selects MXFP8 on supported Blackwell GPUs. That path requires every count to be divisible by 128; an ordinary uneven route hits a CUDA trap instead of completing training. Pad each expert's tokens before the grouped calls and remove that padding when combining results, or keep this example out of MXFP8 until padding is added.

Knowledge Base Used:

Comment on lines +79 to +81
- attention: heads, key/value heads, head size, RoPE base;
- which projections have a bias;
- for mixture-of-experts (MoE) models: the number of experts and the top-k.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Mistral loses its attention window. The attention checklist omits sliding_window, and the recommended Llama template drops HF's mask without setting TE's window_size. For Mistral checkpoints that use local attention, following this template allows tokens to attend beyond the configured window and changes the model's outputs. Require checking the source model's window settings and passing the matching window_size to TE.

Knowledge Base Used: PyTorch Transformer layers

@sbhavani

Copy link
Copy Markdown
Collaborator

Could we consider a weight mapping check? For every source param and destination, reconstruct the HF tensors from the converted weights, and compare them exactly. This should catch missing weights and swapped ordering

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
@faradawn

Copy link
Copy Markdown
Contributor Author

Hi @sbhavani , I've added a "check_weight_mapping.py" to ensure weight mapping is correct.

Comment on lines +123 to +124
if ignore and re.search(ignore, name):
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Ignored weights still fail

--ignore skips the checkpoint weight but still requires its converted TE rows to be claimed. For example, ignoring mlp.gate_proj.weight leaves its half of fc1_weight unclaimed, so a correct conversion still fails. Account for ignored weights before checking for leftover TE rows.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants