Missing sentencepiece model?
#3
by tombbbb - opened
When calling:
tokenizer = T5Tokenizer.from_pretrained('ElnaggarLab/ankh-base')
I get:
File "/home/tbosc/miniconda3/envs/huggingface_ft/lib/python3.10/site-packages/sentencepiece/__init__.py", line 961, in Load
return self.LoadFromFile(model_file)
File "/home/tbosc/miniconda3/envs/huggingface_ft/lib/python3.10/site-packages/sentencepiece/__init__.py", line 316, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string
Google tells me it's a missing sentencepiece model (spiece.model). Indeed this file seems to be present in other repos.
This will work fine:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('ElnaggarLab/ankh-base')