compat with transformers >= 4.40 and tokenizers >= 0.19

[joprice]

Question

This is probably a known issue, as I'm aware that this project lags a bit behind the fast changes being made in the python transformers library, but I wanted to document a specific compatibility issue I hit:

Tokenizers 0.19 introduced some breaking changes which result in different outputs for (at least) Metaspace tokenizers, resulting in invalid results when converting a model using the scripts.convert script with newer transformers version. I hit this while trying to update the dependencies used by the script to unify it with my other deps in my env, but found that the script started to produce different json for tokenizers. In tokenizer.json, the pre_tokenizers and decoder appear now with a split field instead of add_prefix_space:

<         "prepend_scheme": "always",
<         "split": true
---
>         "add_prefix_space": true,
>         "prepend_scheme": "always"

Breaking changes:

• huggingface/transformers@8e5f76f
• Refactor metaspace tokenizers#1476

[xenova]

Thanks for reporting! Do you have an example model on the HF hub (and example input text) for which transformers.js now produces an incorrect tokenization? That would help a ton!

[joprice]

I've been using "qanastek/pos-french-camembert" and almost any phrase was failing when I converted the model to onnx with the newer transformers version. One short phrase was "J'ai fini". With the pre-19 tokenizers, you get the correct parts of speech

  ("J" "PPER1S")
  ("'", "ADV")
  ("ai", "AUX")
  ("fini", "VPPMS")

With the latest, you get all the entities as MOTINC, which is "unknown words" https://github.com/qanastek/ANTILLES/blob/a860b50a31be9fe81f4b20a1914e84a4f4c7e910/README.md#new-tags-based-on-lia_tagg.

If you need more information, I can make a quick repro repo.

[baowendin]

@xenova Llama-2-7b-chat-hf can be used to re-produce this problem, you can download the repo and remove tokenizer.json(for going to the convert path)
with code from below:

from transformers import AutoTokenizer

if __name__ == '__main__': 
    tokenzier = AutoTokenizer.from_pretrained("/path/to/dir", trust_remote_code=True)
    encoded = tokenzier.encode("hello</s>ASSISTANT:")
    decoded = tokenzier.decode(encoded)
    print("encoded: ", encoded)
    print("decoded: ", decoded.replace("\n", "\\n"))

when in transformers==4.39.3, we can get result as below and this is the default behavior that space is added in prefix:

encoded:  [1, 22172, 2, 319, 1799, 9047, 13566, 29901]
decoded:  <s> hello</s> ASSISTANT:

but in transformers==4.44.0, the result would be:

encoded:  [1, 22172, 2, 22933, 9047, 13566, 29901]
decoded:  <s> hello</s>ASSISTANT:

both the tokens and decoded text are changed
if there's any way to get the same result?