Add Qwen3-TTS minimal adapter inference (5/5)

README.md

@@ -405,6 +405,7 @@ foundation for building customizable, enterprise-grade training services.
 
 | Component Type | Component Link                                                                                           | Component Function                                                                      | Author              |
 | -------------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ------------------- |
+| Cookbook | [Qwen3-TTS LoRA](cookbook/qwen3_tts/README.md) | Single Ascend NPU training, explicit data splits and update-boundary resume | [Wallbreazzz](https://github.com/Wallbreazzz) |
 | Patch          | [qwen3_moe_transformers4_patch](https://www.modelscope.cn/models/twinkle-kit/qwen3_moe_transformers4_patch) | Fixes Qwen3 MoE model hang issue during FSDP2 training, effective for transformers==4.x | ModelScope Official |
 
 ## Contributions

cookbook/qwen3_tts/.gitignore

@@ -0,0 +1,3 @@
+outputs/
+__pycache__/
+*.py[cod]

cookbook/qwen3_tts/GENERATION.md

@@ -0,0 +1,29 @@
+# Generate speech with a trained adapter
+
+Run in the cookbook directory with the existing local model and optional dependencies:
+
+```bash
+ASCEND_RT_VISIBLE_DEVICES=0 python scripts/generate.py \
+  --model_path /models/Qwen3-TTS-12Hz-1.7B-Base \
+  --checkpoint /data/run/checkpoint-update-70 \
+  --reference_audio /data/reference.wav \
+  --text "A new sentence to synthesize." \
+  --output /data/generated.wav
+```
+
+Omit `--checkpoint` to use the base model. The output must be new. `--seed`
+(default 42) controls sampling, and `--max_new_tokens` (default 256) bounds
+length. The reference must be finite mono 16/24 kHz audio from the intended
+speaker. Prefer the reference used to prepare training data.
+
+Auto language conditioning and speaker-only cloning match this cookbook's
+training prefix. Sampling uses the pinned official implementation's defaults;
+TTS runs in NPU bf16 and the codec in CPU fp32. Models load locally without
+a download fallback. WAV values are saved without normalization or clipping.
+A valid WAV does not guarantee correct speech: listen for omissions, repetition
+and truncation at the token limit.
+
+For the optional real-model inference smoke test, provide `MODEL_PATH` and
+`CHECKPOINT_PATH`, then run `python tests/entrypoint/test_inference_npu.py`.
+It checks finite audio, not quality. HTML comparison pages, human-score
+collection and project acceptance workflows are outside this cookbook.

cookbook/qwen3_tts/README.md

@@ -0,0 +1,158 @@
+# Qwen3-TTS LoRA training on one Ascend NPU
+
+This cookbook connects the Qwen3-TTS 12Hz Base model to Twinkle's local
+Transformers training API. It provides explicit train/dev preparation,
+single-sample micro-batches, LoRA on the Talker and Code Predictor, accumulation
+with a final short window, and restart at completed optimizer updates.
+
+It is a single-card reference implementation, not a quality-tuned training
+recipe. There are no changes to Twinkle core or its default dependencies.
+A minimal adapter inference example is documented in [GENERATION.md](GENERATION.md).
+
+## Installation and supported scope
+
+Run from the repository root with an existing Ascend PyTorch environment:
+
+```bash
+pip install -e .
+pip install -r cookbook/qwen3_tts/requirements.txt
+cd cookbook/qwen3_tts
+export MODEL_PATH=/models/Qwen3-TTS-12Hz-1.7B-Base
+export ASCEND_RT_VISIBLE_DEVICES=0
+```
+
+The model directory must already contain the complete Base weights, config,
+text tokenizer, processor and `speech_tokenizer/`. Preparation and training are
+offline: missing assets cause an error, not a download. No virtual environment
+or model download is created by these commands. Select one physical card with
+`ASCEND_RT_VISIBLE_DEVICES`; `npu:0` is relative to that selection.
+
+The development baseline used Twinkle `5337cc3`, Qwen3-TTS `022e286`, Python
+3.12, torch 2.10.0, torch_npu 2.10.0.post2, CANN 9.1.0 and an Ascend 910B.
+The optional requirements pin the Qwen implementation and its Transformers
+4.57.3 / accelerate 1.12.0 dependencies. They are not installed for ordinary
+Twinkle users. Match torch/torch_npu to your Ascend image before installing
+these extras. Codec and reference mel extraction use CPU fp32; the TTS model
+uses NPU bf16 with eager attention. Multi-card, CUDA, padding-free execution
+and other Qwen3-TTS model variants are not covered by the current evidence.
+
+## Prepare an existing split
+
+Provide two UTF-8 JSONL files. Each row has `id`, `text`, `audio`, `group_id`
+and `speaker_id`. Paths are relative to the input JSONL file. For example:
+
+```json
+{"id":"train-001","text":"A training utterance.","audio":"audio/train-001.wav","group_id":"session-a","speaker_id":"speaker-a"}
+```
+
+Use a different held-out group and text in the dev file. Audio must be finite
+mono 16 or 24 kHz; 16 kHz is resampled to 24 kHz. The preparer requires one
+speaker, unique IDs, nonempty train/dev sets and a reference chosen from train.
+It rejects cross-split group, normalized text, file, decoded PCM and resampled
+PCM duplicates. Group identifiers still need to reflect the real recording
+source; these checks do not detect all near-duplicates.
+
+```bash
+python scripts/prepare_split_manifest.py \
+  --model_path "$MODEL_PATH" \
+  --train_jsonl /data/splits/train.jsonl \
+  --dev_jsonl /data/splits/dev.jsonl \
+  --reference_id train-001 \
+  --output_dir /data/derived/qwen3-tts
+```
+
+The output directory must be new. The script preserves the assigned split,
+text and order; it never silently resplits or edits source audio. Output rows
+contain `id`, `text`, integer `audio_codes` of shape `[T,16]` in `[0,2048)`,
+and `ref_mel_path`. One train-reference mel `[F,128]` is shared by both splits.
+`provenance.json` records input/output hashes and the source mapping.
+The dev manifest is prepared for downstream evaluation and is **not** consumed
+by this training loop. No dev loss, early stopping or quality score is implied.
+
+## Train and resume
+
+```bash
+python scripts/train_qwen3_tts_lora.py \
+  --model_path "$MODEL_PATH" \
+  --train_jsonl /data/derived/qwen3-tts/train.jsonl \
+  --output_dir outputs/run-1 \
+  --epochs 3 --grad_accum 4 --shuffle --seed 42 \
+  --lr 1e-4 --lr_decay 1.0 --lora_r 8 --lora_dropout 0.1 \
+  --save_interval 10
+```
+
+These are example hyperparameters, not a convergence recommendation. The
+default learning rate stays constant. `--lr_decay` optionally supplies StepLR
+gamma per optimizer update; the short resume regression explicitly uses 0.8
+to exercise scheduler restoration. LoRA targets `q_proj`/`v_proj` in both
+branches, with alpha equal to twice the rank. All base parameters are frozen.
+
+Micro-batch size is one. The loop owns accumulation windows and keeps
+Twinkle's group accumulation at one; `clip_grad_and_step` normalizes by the
+actual number of micro-batches, including an epoch's short final window.
+The speaker encoder runs without gradients. The Talker loss uses its internal
+temporal shift; residual-codebook logits are supervised against corresponding
+codebooks 1–15 without another shift along the codebook axis. Tests check
+last-head gradients and the official residual assembly interface. This loss
+choice is not a claim about how official pretrained weights were trained.
+
+For a controlled stop, add `--stop_after_updates 1`. Resume to a **new** output
+directory with the same training options:
+
+```bash
+python scripts/train_qwen3_tts_lora.py \
+  --model_path "$MODEL_PATH" \
+  --train_jsonl /data/derived/qwen3-tts/train.jsonl \
+  --output_dir outputs/run-2 --resume_from outputs/run-1/checkpoint-update-1 \
+  --epochs 3 --grad_accum 4 --shuffle --seed 42 \
+  --lr 1e-4 --lr_decay 1.0 --lora_r 8 --lora_dropout 0.1 \
+  --save_interval 10
+```
+
+Checkpoints include adapter, optimizer, scheduler, RNG and loop progress.
+Resume checks content fingerprints of model/manifest/reference mel, sample
+plan and settings; it fails on mismatch instead of silently starting over.
+Model hashing reads the local weight files and may take time. Only complete,
+trusted checkpoints produced by this workflow should be resumed. Saving is
+at update boundaries; recovery from arbitrary interruption during a checkpoint
+write is not guaranteed. Inspect `trace.jsonl`, `run_config.json` and
+`result.json` for the actual update count and checkpoint path.
+
+## Tests and evidence boundaries
+
+With the optional dependencies installed:
+
+```bash
+python tests/run.py
+MODEL_PATH="$MODEL_PATH" python tests/run.py --npu
+```
+
+The first command runs data, forward/loss, projection, manifest, preparation
+and accumulation contracts. It includes official mel/residual-interface tests
+but uses small CPU networks, not the pretrained TTS model. The second also
+loads the real local model for a LoRA update/save/reload, public split
+preparation-to-training, and continuous versus independent-process resume.
+The two tiny LibriSpeech fixtures are attributed under
+`tests/fixtures/librispeech/`; they are regression fixtures, not an independent
+quality benchmark. Tests use temporary outputs and never require user data.
+
+For local CPU development without Twinkle or Qwen installed, install torch,
+Transformers, NumPy, scipy and soundfile, then run `python tests/run.py
+--isolated`. This mode explicitly substitutes Twinkle infrastructure and skips
+two official-package checks. It is not NPU or native Twinkle validation.
+
+The CPU suite checks data, losses and boundary contracts. The opt-in NPU
+suite exercises local-model parameter updates, reload and process recovery.
+Passing these tests does not establish voice quality or convergence on a new
+dataset. Select training settings and checkpoints using held-out speech.
+
+## Integration boundaries
+
+The local `tts/` package is added to the script path; it is not registered as
+a new global Twinkle task. The subclass reuses forward/backward, optimizer,
+clipping, saving and resume APIs. It currently overrides
+`_construct_default_optimizer_group` for consistent tokenizer/processor/loss
+defaults and calls `_lazy_wrap_model` when inspecting/loading wrapped weights.
+These internal dependencies are isolated here and should be reviewed when
+Twinkle changes. A public generic TTS abstraction should only be added if
+maintainers identify a reusable need; this cookbook does not add one prematurely.

cookbook/qwen3_tts/requirements.txt

@@ -0,0 +1,8 @@
+accelerate==1.12.0
+numpy>=2.0,<2.3
+peft==0.17.1
+qwen-tts @ git+https://github.com/QwenLM/Qwen3-TTS.git@022e286b98fbec7e1e916cb940cdf532cd9f488e
+safetensors
+scipy
+soundfile
+transformers==4.57.3

cookbook/qwen3_tts/scripts/audio_features.py

@@ -0,0 +1,42 @@
+"""Official 24 kHz mel extraction and 16-codebook audio encoding."""
+
+import numpy as np
+
+MEL_DIM = 128
+MEL_HOP = 256
+MEL_WIN = 1024
+MEL_FMAX = 12000.0
+TARGET_SR = 24000
+
+def extract_mel(wav_24k: np.ndarray) -> np.ndarray:
+    """使用官方 mel 算法，返回单条 [F,128]，不使用替代 fbank。"""
+    import torch
+    from qwen_tts.core.models.modeling_qwen3_tts import mel_spectrogram
+
+    wav = np.asarray(wav_24k, dtype=np.float32)
+    if wav.ndim != 1:
+        raise ValueError('Expected mono waveform [samples]')
+    with torch.no_grad():
+        mel = mel_spectrogram(
+            torch.from_numpy(wav).unsqueeze(0), n_fft=MEL_WIN,
+            num_mels=MEL_DIM, sampling_rate=TARGET_SR, hop_size=MEL_HOP,
+            win_size=MEL_WIN, fmin=0, fmax=MEL_FMAX,
+        ).transpose(1, 2)
+    return mel[0].cpu().numpy()
+
+def encode_audio_codes(wav_24k: np.ndarray, tokenizer) -> np.ndarray:
+    """单条输入必须返回一个 [T,16]；兼容列表、二维张量和单批张量。"""
+    result = tokenizer.encode(wav_24k, sr=TARGET_SR)   # wav_24k 已升采样到 24kHz
+    codes = result.audio_codes
+    if isinstance(codes, (list, tuple)):
+        if len(codes) != 1:
+            raise ValueError('Single waveform must return exactly one audio_codes entry')
+        codes = codes[0]
+    if hasattr(codes, 'cpu'):
+        codes = codes.cpu().numpy()
+    codes = np.asarray(codes)
+    if codes.ndim == 3 and codes.shape[0] == 1:
+        codes = codes[0]
+    if codes.ndim != 2 or codes.shape[0] == 0 or codes.shape[1] != 16:
+        raise ValueError(f'Expected nonempty audio_codes [T,16], got {codes.shape}')
+    return codes.astype(np.int64)

cookbook/qwen3_tts/scripts/generate.py

@@ -0,0 +1,96 @@
+"""Generate one WAV from a local Qwen3-TTS Base model and optional LoRA adapter."""
+import argparse
+import os
+from pathlib import Path
+import sys
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+
+def parse_args(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--model_path', required=True)
+    parser.add_argument('--checkpoint', help='Optional adapter directory; omit for the base model')
+    parser.add_argument('--reference_audio', required=True)
+    parser.add_argument('--text', required=True)
+    parser.add_argument('--output', required=True, help='New output WAV path')
+    parser.add_argument('--seed', type=int, default=42)
+    parser.add_argument('--max_new_tokens', type=int, default=256)
+    args = parser.parse_args(argv)
+    if not args.text.strip() or args.max_new_tokens < 2:
+        parser.error('Nonempty text and max_new_tokens >= 2 required')
+    if Path(args.output).suffix.lower() != '.wav':
+        parser.error('--output must end in .wav')
+    return args
+
+
+def write_wave(path, wave, sample_rate):
+    import numpy as np
+    import soundfile as sf
+    wave = np.asarray(wave, dtype=np.float32)
+    if wave.ndim != 1 or not wave.size or not np.isfinite(wave).all():
+        raise ValueError('Expected a finite nonempty mono waveform')
+    if sample_rate <= 0 or not np.any(wave):
+        raise ValueError('Invalid sample rate or silent waveform')
+    # Preserve levels without clipping; never overwrite an earlier result.
+    with Path(path).open('xb') as output:
+        sf.write(output, wave, sample_rate, format='WAV', subtype='FLOAT')
+
+
+def main():
+    args = parse_args()
+    output = Path(args.output)
+    if output.exists():
+        raise FileExistsError(output)
+    os.environ['HF_HUB_OFFLINE'] = '1'
+    os.environ['TRANSFORMERS_OFFLINE'] = '1'
+    import random
+    import numpy as np
+    import torch
+    import torch_npu  # noqa: F401
+    import twinkle
+    from twinkle import DeviceMesh
+    from transformers import AutoProcessor
+    from qwen_tts import Qwen3TTSModel as OfficialModel
+    from tts.qwen3.runtime import load_twinkle_model, ADAPTER
+    from prepare_split_manifest import read_audio
+
+    torch.npu.set_device(0)
+    mesh = DeviceMesh.from_sizes(fsdp_size=1, dp_size=1)
+    twinkle.initialize(mode='local', global_device_mesh=mesh)
+    path = Path(args.model_path).resolve(strict=True)
+    model = load_twinkle_model(path, mesh)
+    if args.checkpoint:
+        model.add_adapter_to_model(ADAPTER, str(Path(args.checkpoint).resolve(strict=True)), is_trainable=False)
+    model._lazy_wrap_model()
+    core = model.strategy.unwrap_model(model.model)
+    if args.checkpoint:
+        core.set_adapter(ADAPTER)
+        core = core.get_base_model()
+    core.requires_grad_(False)
+    core.eval()
+    core.speech_tokenizer.model.to(device='cpu', dtype=torch.float32).eval()
+    core.speech_tokenizer.device = torch.device('cpu')
+    processor = AutoProcessor.from_pretrained(str(path), local_files_only=True, fix_mistral_regex=True)
+    official = OfficialModel(core, processor, generate_defaults=core.generate_config)
+    wave, _ = read_audio(Path(args.reference_audio).resolve(strict=True))
+    random.seed(args.seed)
+    np.random.seed(args.seed)
+    torch.manual_seed(args.seed)
+    torch.npu.manual_seed(args.seed)
+    with torch.inference_mode():
+        prompt = official.create_voice_clone_prompt(ref_audio=(wave, 24000), x_vector_only_mode=True)
+        waves, sample_rate = official.generate_voice_clone(
+            text=args.text, language='Auto', voice_clone_prompt=prompt,
+            do_sample=True, subtalker_dosample=True,
+            non_streaming_mode=True, max_new_tokens=args.max_new_tokens)
+    if len(waves) != 1:
+        raise ValueError('Expected one generated waveform')
+    output.parent.mkdir(parents=True, exist_ok=True)
+    write_wave(output, waves[0], sample_rate)
+    print(f'Wrote {output}; listen for content accuracy and possible truncation.')
+
+
+if __name__ == '__main__':
+    main()