alibayram/turkish_tiktokenizer at 9d94b4f4aeb879dcd9d29e92e4ddc4c9bd9c6891

10.6 kB

1 contributor

Enhance README with additional sections for Turkish Tokenizer, including a new description, example texts, and dependencies for better clarity and user guidance.

9d94b4f 4 months ago

.gitattributes
1.52 kB

initial commit 11 months ago
.gitignore
15 Bytes

Update REQUIRED_FILES to reference kokler_v07.json instead of kokler_v05.json 10 months ago
README.md
2.13 kB

Enhance README with additional sections for Turkish Tokenizer, including a new description, example texts, and dependencies for better clarity and user guidance. 4 months ago
app.py
6.86 kB

Refactor Turkish Tokenizer implementation by removing unused files and integrating TurkishTokenizer class; update requirements to include turkish-tokenizer package. 4 months ago
requirements.txt
24 Bytes

Refactor Turkish Tokenizer implementation by removing unused files and integrating TurkishTokenizer class; update requirements to include turkish-tokenizer package. 4 months ago