Upload tokenizer e7c5948 verified
Sambit Kumar Barik commited on
How to use NaiveAttention/demo-tokenizer with Transformers:
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("NaiveAttention/demo-tokenizer", device_map="auto")