Adding new non-latin tokens to the T5 tokenizer creates unnecessary whitespaces

[sl5035]

System Info

• transformers version: 4.27.4
• Platform: Linux-5.4.0-99-generic-x86_64-with-glibc2.29
• Python version: 3.8.10
• Huggingface_hub version: 0.15.1
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Tensorflow version (GPU?): 2.12.0 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

@Arthur

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Hi, I am trying to add tokens to the Pix2structProcessor which uses the T5 tokenizer. I have added non-latin characters (Korean) to the tokenizer using the below script so the len(processor.tokenizer.get_vocab()) becomes 65536 (from its original 50344).

processor = Pix2StructProcessor.from_pretrained("deplot_models/deplot_base_model/", is_vqa=True)
model = Pix2StructForConditionalGeneration.from_pretrained(
    "deplot_models/deplot_base_model/", is_vqa=True
)

with open("data/full_vocab.txt", "r+") as f:
    full_v = [v.strip("\n") for v in f.readlines()]
new_t = full_v[50345:]

processor.tokenizer.add_tokens(new_t)

print("Processor loaded!")

The problem arises when I try to tokenize some korean sentences using the extended tokenizer:
processor.tokenizer.tokenize('토크나이저 테스트 중입니다') outputs ['▁', '토', '크', '나이', '▁저', '▁', '테', '스트', '▁중', '입니다'].

The fifth token should be '저' instead of '▁저' (with the underscore) since I've added both of them to the vocab, but instead the tokenizer outputs the wrong version.

TL;DR

1. Extend the T5 tokenizer using non-latin characters.
2. Tokenize a sentence.
3. Error happens.

Expected behavior

The original T5 tokenizer (without extending vocabs) outputs correctly:
['▁', '토', '크', '나', '이', '저', '▁', '테', '스트', '▁중', '입니다']

[xenova]

Hi @sl5035 👋 I see you're using quite an outdated version of transformers (from a ~6 months ago). Could you try upgrade to see if that solves your problem?

Edit: It may have been fixed already at b52a03c (2 months ago)

[sl5035]

Hey, @xenova

Even after upgrading transformers to 4.33.1 unfortunately that didn't solve my problem

[sl5035]

Solved this problem by directly adding the tokens from the tokenizer.json file. It seems like it's an issue from the add_tokens() method.

[ArthurZucker]

Hey! I'll see what I can do! I would recommend trying on main as well

[sl5035]

Thanks, @ArthurZucker !

[ArthurZucker]

Did it work ?

[sl5035]

Hey, @ArthurZucker
Just tried using the transformers 4.34.0.dev0 version, but it still outputs the same.

[ArthurZucker]

Ok! Thanks for testing. I suspect this has to do with the non legacy behaviour.

>>> processor = Pix2StructProcessor.from_pretrained("deplot_models/deplot_base_model/", is_vqa=True, legacy = False, use_fast = False)
>>> processor.tokenizer.add_tokens(['크','나','이','저'])
>>> processor.tokenizer.tokenize('토크나이저 테스트 중입니다')
['▁', '토', '크', '나', '이', '저', '테', '스트', '▁중', '입니다']

The tokens are not added as already part of the vocab:

        2558: AddedToken("이", rstrip=True, lstrip=True, single_word=False, normalized=True, special=False),
        9772: AddedToken("나", rstrip=True, lstrip=True, single_word=False, normalized=True, special=False),
        24470: AddedToken("크", rstrip=True, lstrip=True, single_word=False, normalized=True, special=False),
        26199: AddedToken("저", rstrip=True, lstrip=True, single_word=False, normalized=True, special=False),

but marked as addedTokens.
In that case since you don't specify the behaviour, it will always strip left and right.
The extra space is related to #25224

[sl5035]

Thanks, @ArthurZucker !
Ran a few tests, seems like setting legacy=False, use_fast=False did the trick but it removed most of the whitespaces after fine-tuning and decoding using model.generate(). I've decided to stick with modifying the pre-tokenizer by setting add_prefix_spaces=False. It seems like this method tokenizes well and produces no error when decoding.

[ArthurZucker]

Cool! Could you elaborate on the spaces that were removed? If it's not expected then it's a bug on our end

[sl5035]

Sorry for the late reply @ArthurZucker ,
Almost all of the whitespaces were removed, regardless of whether it is a separate token or not.

Prediction:
TITLE | 36세 남자가콧등에 통증을동반한 피부발진을주소로<0x0A> | 나사콜라이리 |활차상 <0x0A> 0 | 0.02 | 0.1 <0x0A> 8 | 0.02 | 0.03 <0x0A> 16 | 0.03 | 0.32 <0x0A> 24 | 0.15 | 0.02 <0x0A> 32 | 0.01 | 0.12 <0x0A> 40 | 0.02 | 0.03 <0x0A> 48 | 0.38 | 0.12 <0x0A> CAPTION |정부는 지난 4월 29일국제개발협력위원회의통합조정기능강화등을내용으로하는국제개발협력기본법 전부개정안이국회에서 통과됨에따라전략수립사업기획발굴사업심사조정점검평가등ODA주기를혁신하는작업을추진하고있다

Target:
TITLE | 36세 남자가 콧등에 통증을 동반한 피부발전을 주소로 <0x0A> | 나사콜라이리 | 활차상 <0x0A> 0 | 0.02 | 0.1 <0x0A> 8 | 0.02 | 0.03 <0x0A> 16 | 0.03 | 0.32 <0x0A> 24 | 0.15 | 0.02 <0x0A> 32 | 0.01 | 0.12 <0x0A> 40 | 0.02 | 0.03 <0x0A> 48 | 0.38 | 0.12 <0x0A> CAPTION | 정부는 지난 4월 29일 국제개발협력위원회의 통합조정기능 강화 등을 내용으로 하는 국제개발협력기본법 전부개정안이 국회에서 통과됨에 따라 전략수립사업 기획발굴사업 심사조정점검평가 등 ODA 주기를 혁신하는 작업을 추진하고 있다

Maybe it's a bug on my side of the code, but this is what I got from the model!

[ArthurZucker]

When decoding you might want to use spaces_between_special_tokens but otherwise I think it is expected. T5 will encode '▁' as '' and remove it. That's why you need to add the ''▁나' token as well making sure it does not strip

[sl5035]

@ArthurZucker ,
Ah, that makes sense. Thanks!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.