Instructions to use pietrolesci/tokenizers with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use pietrolesci/tokenizers with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("pietrolesci/tokenizers", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Upload folder using huggingface_hub
Browse files- raw_tok_path.txt +1 -0
raw_tok_path.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
/home/pl487/primer/outputs/tok_train/bpe_2025-09-20T17-01-07
|