prelington
/

CodexTrouter

Text Classification

Model card Files Files and versions

CodexTrouter / tokenizer_config.json

prelington's picture

Create tokenizer_config.json

7b3a2af verified 3 months ago

history blame contribute delete

122 Bytes

	{
	"do_lower_case": false,
	"unk_token": "<unk>",
	"pad_token": "<pad>",
	"bos_token": "<s>",
	"eos_token": "</s>"
	}