Kabyle Stanza Tokenizer

Sentence tokenizer for Kabyle (Taqbaylit) language, trained on Tatoeba corpus designed to be used on MiniSBD.

Model Details

Property	Value
Language	Kabyle (`kab`)
Type	Sentence tokenizer
Architecture	CNN + BiLSTM
Training data	Tatoeba Kabyle sentences (~789K sentences)
Dev F1	99.19%
Token F1	99.96%
Sentence F1	98.43%
ONNX size	0.62 MB
Vocab size	223 characters

Files

kab.onnx — ONNX runtime model
kab.pt — PyTorch checkpoint
vocab.json — Character vocabulary (223 entries)
config.json — Model hyperparameters
tokenizer_config.json — HF tokenizer config

Usage

With Stanza (PyTorch)

import stanza
nlp = stanza.Pipeline(
    lang="kab", 
    processors="tokenize", 
    tokenize_model_path="kab.pt"
)
doc = nlp("Amcic ha-t-an deg uxxam-nneɣ. Teciḍ fell-as?")
for sent in doc.sentences:
    print(sent.text)
# Amcic ha-t-an deg uxxam-nneɣ.
# Teciḍ fell-as?

With ONNX Runtime

import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("kab_tokenizer.onnx")
# units: (batch, seq_len) int64 — char IDs from vocab.json
# features: (batch, seq_len, 5) float32 — Stanza features
units = np.zeros((1, 100), dtype=np.int64)
features = np.zeros((1, 100, 5), dtype=np.float32)
outputs = sess.run(None, {"units": units, "features": features})
# outputs[0] shape: (batch, seq_len, 3) — logits for B/I/O

Tokenization Examples

Input	Tokens
`Ad tseddumt ɣer Taskriwt.`	`['Ad', 'tseddumt', 'ɣer', 'Taskriwt', '.']`
`Aweḍ ɣer Tezmalt.`	`['Aweḍ', 'ɣer', 'Tezmalt', '.']`
`Tettawḍem ɣer Kendira.`	`['Tettawḍem', 'ɣer', 'Kendira', '.']`
`Efk-asen tizwal-nni.`	`['Efk-asen', 'tizwal-nni', '.']`
`Melmi ara ad d-taɣeḍ lmitra?`	`['Melmi', 'ara', 'ad', 'd-taɣeḍ', 'lmitra', '?']`

Character Set

The tokenizer handles standard Kabyle Latin characters including:

ɛ / Ɛ (open e)
ɣ / Ɣ (voiced velar fricative)
ṭ, ḍ, č, ǧ (emphatic and palatal consonants)
Standard ASCII + punctuation

Citation

If you use this model, please cite:

Tatoeba project: https://tatoeba.org
Stanza: Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. ACL.

License

Apache-2.0

Downloads last month: -