Accept the OncillaAI Chatterbox Farsi Non-Commercial License to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is licensed for non-commercial use only under the OncillaAI Chatterbox Farsi Non-Commercial License v1.0 (see LICENSE). It may not be used, nor may its outputs be used, to train, distil or improve any other model. Public use must show "Powered by OncillaAI Chatterbox Farsi". For commercial use, contact mehdi@oncillaai.com.

OncillaAI Chatterbox Farsi

A Persian (Farsi) text-to-speech model: natural, expressive Persian speech from ordinary Persian text, in any voice you give it a few seconds of. It is a fine-tune of the multilingual Chatterbox model by Resemble AI.

Reads ordinary Persian script with punctuation; pauses at commas, a falling tone at full stops, rising at «؟».
Two built-in voices (female, male) and zero-shot voice cloning from a 5-10 s clip.
About 0.5 B parameters in the fine-tuned text-to-speech-token model; speech decoder and voice encoder from the base.
Runs on an NVIDIA GPU: close to real time on a T4, about 2.5x faster than real time on an H100.

Samples

News

Female	Male

“بر اساس گزارش سازمان هواشناسی، بارش باران و برف در بیشتر استان‌های شمالی و غربی کشور تا پایان هفته ادامه خواهد داشت. از رانندگان خواسته شده است که در جاده‌های کوهستانی با احتیاط بیشتری حرکت کنند و زنجیر چرخ همراه داشته باشند.”

Instructions

Female	Male

“پیش از شروع به کار با این دستگاه، ابتدا مطمئن شوید که کابل برق به‌درستی به پریز وصل شده است. سپس دکمه‌ی روشن و خاموش را برای سه ثانیه نگه دارید تا چراغ سبز روشن شود. اگر چراغ قرمز شد، دستگاه را خاموش کنید، پنج دقیقه صبر کنید و دوباره امتحان کنید. در صورت تکرار مشکل، با بخش پشتیبانی تماس بگیرید.”

Story

Female	Male

“پیرمرد از پنجره‌ی کوچک اتاق به کوچه نگاه کرد. باران ریزی می‌بارید و چراغ‌های زرد خیابان روی آسفالت خیس می‌درخشیدند. آهی کشید و با خودش گفت که این شهر دیگر شهر روزهای جوانی‌اش نیست.”

Names and technology

Female	Male

“شرکت گوگل در ماه گذشته از نسخه‌ی تازه‌ای از مدل هوش مصنوعی خود رونمایی کرد. مدیرعامل این شرکت، ساندر پیچای، در سخنرانی خود در شهر سانفرانسیسکو گفت که این مدل می‌تواند متن، تصویر و صدا را با هم بفهمد و در بیش از صد زبان، از جمله فارسی، پاسخ دهد.”

Conversational, excited

Female	Male

“وای، باورم نمی‌شه! جدی می‌گی؟ یعنی واقعاً قبول شدم؟ یه بار دیگه نگاه کن، شاید اشتباه خوندی. نه، نه، درسته، اسم منه! خدایا، این همه سال زحمت کشیدم، حالا بالاخره نتیجه شو دیدم. باید همین الان به بابا زنگ بزنم.”

Conversational

Female	Male

“ببین، من نمی‌گم که تو اشتباه می‌کنی. فقط می‌گم قبل از اینکه تصمیم آخرتو بگیری، یه بار دیگه همه چیزو خوب بسنج. اگه اون کار جدید واقعاً چیزیه که می‌خوای، منم پشتتم، ولی دوست ندارم بعداً از یه تصمیم عجولانه پشیمون بشی.”

Quick start

pip install chatterbox-tts "setuptools<81"   # setuptools<81: the watermarker's import needs pkg_resources

import shutil
from pathlib import Path

import torchaudio
from huggingface_hub import hf_hub_download, snapshot_download
from chatterbox.mtl_tts import ChatterboxMultilingualTTS

REPO = "mehdi-hf/chatterbox-farsi"
base = Path(snapshot_download("ResembleAI/chatterbox", allow_patterns=[
    "ve.pt", "s3gen.pt", "conds.pt", "grapheme_mtl_merged_expanded_v1.json", "Cangjie5_TC.json"]))
ckpt = Path("chatterbox_farsi"); ckpt.mkdir(exist_ok=True)
for f in base.iterdir():
    shutil.copy(f, ckpt / f.name)
shutil.copy(hf_hub_download(REPO, "t3_fa.safetensors"), ckpt / "t3_mtl23ls_v2.safetensors")  # the loader's file name
model = ChatterboxMultilingualTTS.from_local(ckpt, "cuda")

voice = hf_hub_download(REPO, "voices/female.wav")   # or voices/male.wav, or a clip of your own
text = "سلام! امروز هوا خیلی خوب است، بیا با هم به پارک برویم."


def generate(text, tries=3, **kw):
    """Now and then the model stops almost at once; take such a cut-off take again."""
    for _ in range(tries):
        wav = model.generate(text, **kw)
        if wav.shape[-1] / model.sr >= len(text) / 40:   # Persian speech runs at ~12-15 characters a second
            break
    return wav


wav = generate(text, language_id="ar", audio_prompt_path=voice)   # Persian uses the "ar" language token
torchaudio.save("out.wav", wav, model.sr)

Tips

Punctuate. The model reads punctuation: end sentences with «.» or «؟», and put a comma where a pause belongs. A comma also stops an unwanted ezafe: «هوا، سرد بود» reads havâ, sard bud.
Force an ezafe where the model misses one by writing it: «کتابِ من», «خانهٔ بزرگ», «هوایِ سرد».
Long text: generate sentence by sentence and join with a short pause; one call holds up to about 40 s.
Settings: the defaults (cfg_weight=0.5, exaggeration=0.5, temperature=0.8) were used for the samples. A lower cfg_weight (0.3) speaks a little slower.
Use an NVIDIA GPU. On Apple Silicon (MPS) the output is noticeably worse, not just slower.
Voice cloning: a clean 5-10 s clip of one speaker, without music, works best.

Model details


Base model	Chatterbox multilingual (Resemble AI, MIT)
Fine-tuned part	T3, the text-to-speech-token model: a Llama-style transformer, 536 M parameters, full fine-tune
Unchanged from the base	S3 speech tokenizer (25 tokens per second), S3Gen decoder (flow matching + HiFi-GAN vocoder), voice encoder
Text input	Persian script read directly through the base model's multilingual grapheme tokenizer: no phonemiser or G2P; punctuation, ZWNJ and ezafe marks («ـِ», «ـٔ») are kept
Language token	`"ar"` (Persian is not one of the base model's languages)
Conditioning	a speaker embedding (256-d) and 150 speech tokens (6 s) from the reference clip; classifier-free guidance on the text
Output	24 kHz mono audio, up to about 40 s (1,000 speech tokens) per call
Weights	`t3_fa.safetensors`, float32, 2.1 GB
Training	checkpoint at step 11,500 of 12,000, chosen by the lowest loss on held-out speakers; AdamW, learning rate 2e-5 with 500 warm-up steps and a cosine decay, about 16k tokens per batch, bf16 mixed precision, text dropped from 10% of examples to keep guidance working; about 1 hour on one NVIDIA H100

Evaluation

35 held-out passages of 8-30 s (news, conversation, science, stories, instructions, numbers, names, formal and colloquial speech), each generated once as a whole, then transcribed by a Persian speech recogniser and compared with the text. Lower is better; the figures include the recogniser's own errors.

Voice	WER	CER
Female (built-in)	12.5%	3.7%
Male (built-in)	16.9%	5.5%

Ezafe (the unwritten linking vowel), rated by a native speaker on 20 test sentences, 2 takes each: 39 of 40 clips right (female), 32 of 32 (male).

Limitations

Ezafe is inferred from context and is right in most noun phrases, but short predicate sentences («مرد خوب است») sometimes get an ezafe they should not have; a comma after the subject fixes it.
Occasionally a take stops early, gains a short sound at the end, or mispronounces a rare word or a foreign name; generating again usually fixes it.
Numbers written in digits are read, but check them in text where every number matters.
Persian is not one of the base model's built-in languages, so the Arabic language token ("ar") is used.

Licence

This model is released under the OncillaAI Chatterbox Farsi Non-Commercial License v1.0 (full text in LICENSE; downloading or using the model means you accept it). In short:

Permitted: research, education, evaluation and personal projects, and sharing the generated audio for non-commercial purposes.
Not permitted: commercial use without a licence; distillation, or using the model or its outputs to train, fine-tune or improve any other model; redistributing the weights or offering them as a service; removing the watermark; the uses listed under Responsible use.
Attribution: public use must show "Powered by OncillaAI Chatterbox Farsi" with a link to this page.
Commercial licences (products, services, paid content, internal business use): contact mehdi@oncillaai.com.

The base model, Chatterbox by Resemble AI, is MIT-licensed; its notice is in NOTICE. Its own components that this model uses unchanged (speech decoder, voice encoder, tokenizer) are downloaded from Resemble AI's repository under that licence.

Responsible use

This model can imitate a real person's voice from a few seconds of audio.

Only clone a voice with that person's permission. Do not use this model to impersonate anyone, to deceive, for fraud or scams, for harassment, or to make speech appear to come from someone who did not say it.
Disclose synthetic speech to listeners where it could be mistaken for a real recording.
Do not present the built-in voices, or any cloned voice, as a real person speaking.
Every output carries Resemble AI's inaudible Perth watermark, applied by the chatterbox-tts package, so generated audio can be identified. Do not remove it.

Credits

Built on Chatterbox by Resemble AI (MIT).

Downloads last month: -

Model tree for mehdi-hf/chatterbox-farsi

Base model

ResembleAI/chatterbox

Finetuned

(95)

this model

mehdi-hf
/

chatterbox-farsi