byte-llm-tts-v1

A byte-level Qwen3 causal LM (38.2M params) finetuned for bilingual text-to-speech (English + Urdu) using the Kanade audio codec.

The base LM was pretrained from scratch on plain text using a 264-token byte-level vocab (256 UTF-8 bytes + 8 specials). For TTS finetuning the vocab was extended to 13,080 tokens by adding: • 16 TTS control specials — <|start_of_speech|>, <|speech|>, <|start_of_text|>, language tags (<|en|>, <|ur|>), prosody tags, and 7 reserved slots for future use. No voice / accent tags — prosody is conveyed by the audio tokens themselves. • 12,800 audio-code tokens <|code_0|> .. <|code_12799|> — Kanade codec content-token indices.

The first 264 ids are byte-identical to the pretrain checkpoint, so all byte-level language knowledge is preserved when the embedding matrix is resized. New rows are initialised with the mean of the existing 264 rows plus small Gaussian noise.

Sequence format

<|start_of_speech|> [lang]
<|start_of_text|> text bytes <|end_of_text|>
<|speech|> audio_code_tokens <|end_of_speech|>

Loss

total_loss = 3.0 × audio_loss + 1.0 × text_loss

with an additional 2× local weight on the <|speech|> and <|end_of_speech|> boundary tokens (missing them = total sequence failure).

Usage

See byte_tokenizer.py, byte_finetune.py, and infer.py in this repo for the full pipeline.