ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.

[candowu]

System Info

4.27.1

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

i test llama in colab here is my code and output:

!pip install git+https://github.com/huggingface/transformers
!pip install sentencepiece

import torch
from transformers import pipeline,LlamaTokenizer,LlamaForCausalLM

device = "cuda:0" if torch.cuda.is_available() else "cpu"
print(device)

tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")

model = LlamaForCausalLM.from_pretrained("decapoda-research/llama-7b-hf")

generator = pipeline(model="decapoda-research/llama-7b-hf", device=device)
generator("I can't believe you did such a ")

ValueError Traceback (most recent call last)
in
7 # tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")
8 # model = LlamaForCausalLM.from_pretrained("decapoda-research/llama-7b-hf")
----> 9 generator = pipeline(model="decapoda-research/llama-7b-hf", device=device)
10 generator("I can't believe you did such a ")

1 frames
/usr/local/lib/python3.9/dist-packages/transformers/models/auto/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
675
676 if tokenizer_class is None:
--> 677 raise ValueError(
678 f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
679 )

ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.

Expected behavior

expect output generated info

[yhifny]

I face the same issue

[amyeroberts]

Hi @candowu, thanks for raising this issue. This is arising, because the tokenizer in the config on the hub points to LLaMATokenizer. However, the tokenizer in the library is LlamaTokenizer.

This is likely due to the configuration files being created before the final PR was merged in.

[yhifny]

I cloned the repo and changed the tokenizer in the config file to LlamaTokenizer
but I got
ValueError: Tokenizer class LlamaTokenizer does not exist or is not currently imported.

[mbehm]

For anybody interested I was able to load an earlier saved model with the same issue using my fork with the capitalization restored. That being said for future it's probably better to try find or save a new model with the new naming.

[amyeroberts]

@yhifny Are you able to import the tokenizer directly using from transformers import LlamaTokenizer ?

If not, can you make sure that you are working from the development branch in your environment using:
pip install git+https://github.com/huggingface/transformers

more details here.

[nadahlberg]

I can import the LlamaTokenizer class, but getting error that from_pretrained method is None. Anyone else having this issue?

[sgugger]

As the error message probably mentions, you need to install sentencepiece: pip install sentencepiece.

[nadahlberg]

Working now. I swear I had sentencepiece, but probably forgot to reset the runtime 🤦 My bad!

[xhinker]

For anybody interested I was able to load an earlier saved model with the same issue using my fork with the capitalization restored. That being said for future it's probably better to try find or save a new model with the new naming.

Thanks, man, your link solved all the problem

[nameless0704]

Hi @candowu, thanks for raising this issue. This is arising, because the tokenizer in the config on the hub points to LLaMATokenizer. However, the tokenizer in the library is LlamaTokenizer.

This is likely due to the configuration files being created before the final PR was merged in.

Change the LLaMATokenizer in tokenizer_config.json into lowercase LlamaTokenizer and it works like a charm.