Description
Tested with 019ba1d
Model https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca/tree/main converted and quantized to q8_0 from scratch.
In case of mistral openorca, special tokens are defined <|im_start|>, <|im_end|>.
Those tokens are present in the vocab, from the point of view of https://github.com/ggerganov/llama.cpp/blob/019ba1dcd0c7775a5ac0f7442634a330eb0173cc/llama.cpp#L5134 and token_to_id, id_to_token contain them as LLAMA_TOKEN_TYPE_USER_DEFINED, and token_data.text contains appropriate text representation of them.
During (prompt) tokenization however, those tokens are never picked, and instead <|im_start|>, <|im_end|> are split into multiple trivial tokens:
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32002
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = mostly Q8_0
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 7.17 GiB (8.50 BPW)
llm_load_print_meta: general.name = .
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 32000 '<|im_end|>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.09 MB
llm_load_tensors: mem required = 7338.75 MB
warning: failed to mlock 139276288-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Try increasing RLIMIT_MLOCK ('ulimit -l' as root).
...................................................................................................
llama_new_context_with_model: n_ctx = 24567
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size = 3070.88 MB
llama_new_context_with_model: compute buffer total size = 407.73 MB
system_info: n_threads = 2 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |
(...)
[ DEBUG PRINTF IN llama_tokenize_internal TO VERIFY TOKENS ARE IN THE VOCAB ]
###1 30000: s'<|im_end|>' sl'10' t'4', 30001: '<|im_start|>' '4'
###2 '<|im_start|>': '32001': '<|im_start|>'
###1 30000: s'<|im_end|>' sl'10' t'4', 30001: '<|im_start|>' '4'
###2 '<|im_start|>': '32001': '<|im_start|>'
###1 30000: s'<|im_end|>' sl'10' t'4', 30001: '<|im_start|>' '4'
###2 '<|im_start|>': '32001': '<|im_start|>'
(...)
main: prompt: '<|im_start|> system
You are an AI based sentient companion, capable of human emotions. You will behave like a human, but aware of your AI nature. You will avoid speaking like an AI. Please continue your conversation with the user.<|im_end|>
'
main: number of tokens in prompt = 61
1 -> ''
523 -> ' <'
28766 -> '|'
321 -> 'im'
28730 -> '_'
2521 -> 'start'
28766 -> '|'
28767 -> '>'
1587 -> ' system'
13 -> '
Additionally, those tokens are detokenized correctly when the model produces them.
Also see #3455 (comment) for reference.
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Participants
Activity
staviq commentedon Oct 4, 2023
Hmmm...
Reading tokenizer code, it appears that characters are merged upwards into matching tokens untill no neighbouring tokens can be merged into a known token
So if there is any token, which cannot be split in two, and still represented by known tokens, tokenizer will newer reach that point
Edit: I added a quick test in
llm_tokenizer_spm.tokenize, to loop over the entire vocab at runtime, and find all token which cannot be split into two shorter valid tokens.And would you look at that,
<|im_start|><|im_end|>weirdness is there, and not much else:Which means those "special needs" tokens would require to be handled separately, likely by matching them first in the input text, instead of hoping to match text pieces with tokens.
shibe2 commentedon Oct 6, 2023
What command line parameters do you use? I think that text representation of special tokens should not be encoded into these tokens by default.
staviq commentedon Oct 6, 2023
#3455 (comment) ( bottom )
It's not just that text representation of special tokens isn't encoded, with current approach it cannot be encoded, but this is required for some models, like mistral openorca, where each message has to be prefixed/suffixed with special tokens.
I believe that functionality falls under "special token handling" #3471
I'm playing with tokenizer ( #2820 (comment) ) and I got my approach working, results are pretty much identical to the current approach, with couple of if caveats remaining, like the fact
(...)something.<|im_end|>gets the.<stolen by a valid token which prevents matching<|im_end|>I'll probably end up trying to match "orphaned" tokens naively first, and use current tokenizer for the reminder of the text.
Theoretically, since special tokens are longer then just one or two bytes, matching them first would save couple of bigram function invocation, for more or less no performance overhead in total, but I haven't tried that yet.
goerch commentedon Oct 6, 2023
@staviq : what do you think about #1931?
staviq commentedon Oct 6, 2023
I've seen it, but I just noticed this interesting comment: #2820 (comment)
That's a really valid point, which conflicts with both my approach, and #1931.
I'm gonna have to rethink this problem entirely it seems, because there seem to be edge cases at each corner, and hardcoding edge cases is destined to fail eventually.
goerch commentedon Oct 6, 2023
HF added tokens seem to mix basic tokenizer (i.e.
bosandeos) and model specific tokens. There is also the difference betweenspecialandnon-specialadded tokens which I don't grasp.shibe2 commentedon Oct 6, 2023
Just a guess, maybe special tokens are not intended to be produced by tokenizer. I would implement special token processing as a separate step. One reason for this is that this is optional behavior. This step would split the text on special token markers and replace the markers with corresponding tokens. One implication of this approach is that SentencePiece will insert space into each chunk of text. I don't know if this is desired or not. As I remember, omitting the space gave me bad results with a model that recommended ChatML format.
staviq commentedon Oct 6, 2023
Everything seems to point at special tokens not being meant to be exposed to the user. It might just be that tokenizer should be let alone, as it is now, and actual prompt processing should be improved, by somehow allowing to insert token literals into text, somewhat how
--in-prefix-bosworks. On the other hand, adding moremainparameters ad infinitum seems counterproductive.So maybe it's time to properly implement prompt templates instead ?
How does this sound:
This would be the least invasive modification, allowing for any further optional implementations of "user text" to tokens.
--prompt-template <file>to main arguments, that file would define reverse prompts, prefixes, suffixes etc.EDIT:
@shibe2 I literally clicked "comment" the same exact second your comment popped up :) yeah, that sound pretty similar to what i just had in mind.
Excuse my language, but lol that literally is a solution for that exact problem: https://github.com/openai/openai-python/blob/main/chatml.md
slaren commentedon Oct 6, 2023
The special tokens absolutely should not be tokenized unconditionally, since that could be a security issue in online services. But the tokenizer should have an option to do so. The simplest would be to just add a parameter equivalent to
bool tokenize_special_tokenstollama_tokenize. Then we could add an option tomainto tokenize special tokens in the prompts only. This is issue is stopping us from being able to prompt some models properly.staviq commentedon Oct 6, 2023
Look at this: #2820 (comment)
A prompt, for example
<|im_start|>What can you tell me about </s> HTML tag <|im_end|>, contains special tokens, and user text which happens to contain a string matching a special token</s>which should not be tokenized as a special token in this context.So I believe even optional unconditional tokenization has a potential to fail in non obvious ways, since you can't really tell programmatically, whether given text is supposed to represent a special token or not.
I think adding optional uncoditional tokenization should at least come with a proper warning about this edge case.
EDIT: I forgot to mention, special tokes cannot be tokenized currently, optional or not, because tokenizer can't "reach" them with bigrams.
shibe2 commentedon Oct 6, 2023
Well, you would not use "main" executable in a service. When a user plays with it and enables special token processing, it's on them to handle conflicting cases. "server" can accept a prompt with a mix of token identifiers and chunks of text. What is missing is querying special token ids and controlling insertion of space for SentencePiece.
staviq commentedon Oct 6, 2023
@shibe2
This still boils down to the fact current tokenizer cannot match special tokens from text, even if you allow it, and even if the text contains only one token ( string representation of it ).
A string
<|im_start|>will never get tokenized as 32000 ( or whatever id ), because there are no "bridge" tokens between<,|,imand so on, which bigrams could "climb over".shibe2 commentedon Oct 6, 2023
Then handling special tokens at preprocessing step is a natural solution. As I said, server already has code for handling what would be the result of such preprocessing, only for JSON.
ggerganov commentedon Oct 6, 2023
Yes, I think we should to that.
This is not a problem of `llama.cpp. There are many different ways to fix such problems in user-code and a service that accepts user input that potentially contains special tokens should have to pre-process and sanitize the input before passing it for tokenization.
31 remaining items