prelington
/

CodexTrouter

Text Classification

Model card Files Files and versions

CodexTrouter / tokenizer.json

prelington's picture

Create tokenizer.json

8a2b266 verified 2 months ago

history blame contribute delete

141 Bytes

	from transformers import AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")
	tokenizer.save_pretrained("./ProTalk/")