Tokenizer loading: This breaks quite a few things in a lot of places

[DevasiaThomas]

https://github.com/huggingface/transformers/blame/638d49983f36af910934b38771b4e55c835c1774/src/transformers/tokenization_utils_base.py#L2253

I was trying to use text-generation-webui and all tokenizer loads, using quite a few loaders that depend on transformers libraries, are broken.

@ArthurZucker Were you just trying to override the special token values in init_kwargs?

I changed the statement as below, locally, to get things to work:
init_kwargs[key] = added_tokens_map.get(key, init_kwargs[key])

I didn't make a PR because I'm not sure what the overall intent is. Prior to this, the flow was different.

Appreciate your attention on this. Thanks :)

[ArthurZucker]

Hey! 🤗 Thanks for reporting this!
Is there anyway you share the errors you were getting.
the goal behind init_kwargs[key] = added_tokens_map.get(init_kwargs[key], init_kwargs[key]) is to make sure that the special_tokens passed as initi_kwargs are replace with their AddedToken version if it exists.
But this should not have broken things (AFAIK) and was tested quite a bit!
Would make sure the version of transformers is >=4.34.1 because I had to do a patch!

Very core issue so I'll be answering quick!

[DevasiaThomas]

@ArthurZucker
It was shared earlier as a link to what I was using :)

here's the issue

I faced the same issue, when trying to load the LLaMa2 model. The issue has others with similar issues on other models: uhashable type dict

the goal behind init_kwargs[key] = added_tokens_map.get(init_kwargs[key], init_kwargs[key]) is to make sure that the special_tokens passed as initi_kwargs are replace with their AddedToken version if it exists. But this should not have broken things (AFAIK) and was tested quite a bit! Would make sure the version of transformers is >=4.34.1 because I had to do a patch!

If that was your intent - then all you need to do is what I did
init_kwargs[key] = added_tokens_map.get(key, init_kwargs[key].
Since you want to use the values for the same keys in AddedToken if they exist (not their values). Unless you actually want to retrieve based on the values from AddedToken instead of keys
I am using the latest transformers.

[ArthurZucker]

Thanks, yeah I have a PR draft at #27099 I'll work on it in priority! Thanks for the detailed report

[DevasiaThomas]

Thanks, yeah I have a PR draft at #27099 I'll work on it in priority! Thanks for the detailed report

Thanks :)

[ArthurZucker]

PR is ready and should fix! 🤗

[Yoline777]

@ArthurZucker Still get unhashable type: 'AddedToken' error, when try tokenizer = CLIPTokenizer.from_pretrained('openai/clip-vit-large-patch14', cache_dir=cache_dir)

[ArthurZucker]

Yep I'll merge this to main soon the PR introduced one regression for a model

[Yoline777]

@ArthurZucker This problem does not seem to be completely resolved, when try
tokenizer = CLIPTokenizer.from_pretrained('openai/clip-vit-large-patch14', cache_dir=cache_dir)
The error in tokenization_utils_base disappeared, but there are still errors in tokenization_utils.

  File "Stable_Diffusion_QCOM/stable_diffusion.py", line 41, in <module>
    tokenizer = CLIPTokenizer.from_pretrained('openai/clip-vit-large-patch14', cache_dir=cache_dir)
  File "/home/linyun/Huggingface_models/transformers/src/transformers/tokenization_utils_base.py", line 2028, in from_pretrained
    return cls._from_pretrained(
  File "/home/linyun/Huggingface_models/transformers/src/transformers/tokenization_utils_base.py", line 2260, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/linyun/Huggingface_models/transformers/src/transformers/models/clip/tokenization_clip.py", line 343, in __init__
    super().__init__(
  File "/home/linyun/Huggingface_models/transformers/src/transformers/tokenization_utils.py", line 368, in __init__
    [token for token in self.all_special_tokens_extended if token not in self._added_tokens_encoder],
  File "/home/linyun/Huggingface_models/transformers/src/transformers/tokenization_utils.py", line 368, in <listcomp>
    [token for token in self.all_special_tokens_extended if token not in self._added_tokens_encoder],
TypeError: unhashable type: 'AddedToken'

[ArthurZucker]

Make sure you are running this on the latest version, I can't reproduce this with:

from transformers import CLIPTokenizer,AutoTokenizer
tokenizer = CLIPTokenizer.from_pretrained('openai/clip-vit-large-patch14'
tokenizer = AutoTokenizer.from_pretrained('openai/clip-vit-large-patch14')

[Yoline777]

@ArthurZucker Thank you for your patience, here is my operation track, it works fine a few months ago.

root@23c6d347b0c3:/home/linyun/Huggingface_models# git clone -b v4.36.2 https://github.com/huggingface/transformers
Cloning into 'transformers'...
remote: Enumerating objects: 175620, done.
remote: Counting objects: 100% (143/143), done.
remote: Compressing objects: 100% (92/92), done.
remote: Total 175620 (delta 69), reused 75 (delta 40), pack-reused 175477
Receiving objects: 100% (175620/175620), 174.69 MiB | 6.00 MiB/s, done.
Resolving deltas: 100% (133029/133029), done.
Note: switching to 'a7cab3c283312b8d4de5df3bbe719971e24f4281'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

root@23c6d347b0c3:/home/linyun/Huggingface_models# python
Python 3.8.10 (default, Nov 22 2023, 10:22:35) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("transformers/src")
>>> from transformers import CLIPTokenizer
>>> cache_dir = "./_data_/cache/huggingface/diffusers"
>>> tokenizer = CLIPTokenizer.from_pretrained('openai/clip-vit-large-patch14', cache_dir=cache_dir)
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████| 905/905 [00:00<00:00, 68.8kB/s]
vocab.json: 100%|███████████████████████████████████████████████████████████████████| 961k/961k [00:00<00:00, 1.02MB/s]
merges.txt: 100%|████████████████████████████████████████████████████████████████████| 525k/525k [00:00<00:00, 994kB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████| 389/389 [00:00<00:00, 190kB/s]
tokenizer.json: 100%|█████████████████████████████████████████████████████████████| 2.22M/2.22M [00:00<00:00, 4.11MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/linyun/Huggingface_models/transformers/src/transformers/tokenization_utils_base.py", line 2028, in from_pretrained
    return cls._from_pretrained(
  File "/home/linyun/Huggingface_models/transformers/src/transformers/tokenization_utils_base.py", line 2260, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/linyun/Huggingface_models/transformers/src/transformers/models/clip/tokenization_clip.py", line 343, in __init__
    super().__init__(
  File "/home/linyun/Huggingface_models/transformers/src/transformers/tokenization_utils.py", line 368, in __init__
    [token for token in self.all_special_tokens_extended if token not in self._added_tokens_encoder],
  File "/home/linyun/Huggingface_models/transformers/src/transformers/tokenization_utils.py", line 368, in <listcomp>
    [token for token in self.all_special_tokens_extended if token not in self._added_tokens_encoder],
TypeError: unhashable type: 'AddedToken'
>>> 

[ArthurZucker]

Could you push this tokenizer to the hub? This way I can have access to the tokenizer_config.json

[Yoline777]

@ArthurZucker Thanks a lot. After I changed token to str(token) in the following 2 places, the error disappeared.

    def _update_trie(self, unique_no_split_tokens: Optional[str] = []):
        for token in self._added_tokens_decoder.values():
            if str(token) not in self.tokens_trie._tokens:
                self.tokens_trie.add(token.content)

        # 4. If some of the special tokens are not part of the vocab, we add them, at the end.
        # the order of addition is the same as self.SPECIAL_TOKENS_ATTRIBUTES following `tokenizers`
        self._add_tokens(
            [token for token in self.all_special_tokens_extended if str(token) not in self._added_tokens_encoder],
            special_tokens=True,
        )

[ArthurZucker]

you change this in your custom tokenization_clip.py and are using remote code if I understood correctly? 🤗

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.