Repository navigation
Conversation
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
[Low impact] The changes affect only users who choose this new recipe-writing skill or run its checker. The PR needs changes before merging because MXFP8 expert splits remain unpadded and Mistral's attention window remains unpreserved. Findings
SummaryAdds a skill for converting Hugging Face models to fused Transformer Engine recipes.
Reviews (4) · Last reviewed commit: "add weight mapping check script to skill" · Reviewed by Greptile |
| split_sizes = torch.bincount(experts.reshape(-1), minlength=num_experts).int() | ||
| permuted, row_id_map = te.moe_permute( # sort tokens by expert | ||
| tokens, experts.to(torch.int32), num_out_tokens=experts.numel(), map_type="index" | ||
| ) | ||
| expert_out = self.experts_ffn(permuted, split_sizes, split_sizes) # one grouped GEMM each |
There was a problem hiding this comment.
MXFP8 experts need padded tokens. The example passes raw expert counts from
torch.bincount to GroupedLinear, but the precision hook selects MXFP8 on supported Blackwell GPUs. That path requires every count to be divisible by 128; an ordinary uneven route hits a CUDA trap instead of completing training. Pad each expert's tokens before the grouped calls and remove that padding when combining results, or keep this example out of MXFP8 until padding is added.
Knowledge Base Used:
| - attention: heads, key/value heads, head size, RoPE base; | ||
| - which projections have a bias; | ||
| - for mixture-of-experts (MoE) models: the number of experts and the top-k. |
There was a problem hiding this comment.
Mistral loses its attention window. The attention checklist omits
sliding_window, and the recommended Llama template drops HF's mask without setting TE's window_size. For Mistral checkpoints that use local attention, following this template allows tokens to attend beyond the configured window and changes the model's outputs. Require checking the source model's window settings and passing the matching window_size to TE.
Knowledge Base Used: PyTorch Transformer layers
|
Could we consider a weight mapping check? For every source param and destination, reconstruct the HF tensors from the converted weights, and compare them exactly. This should catch missing weights and swapped ordering |
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
|
Hi @sbhavani , I've added a "check_weight_mapping.py" to ensure weight mapping is correct. |
| if ignore and re.search(ignore, name): | ||
| continue |
There was a problem hiding this comment.
--ignore skips the checkpoint weight but still requires its converted TE rows to be claimed. For example, ignoring mlp.gate_proj.weight leaves its half of fc1_weight unclaimed, so a correct conversion still fails. Account for ignored weights before checking for leftover TE rows.
Skill: fusions like LayerNorm+QKV, LayerNorm+MLP,
TransformerLayerandGroupedLinearMoE experts, with the checkpoint weight mapping.Eval: Llama, Gemma and Mixtral