server : display token probabilities in the UI

[jhen0409]

#2423

This is a simple implementation for probabilities of llama response.

It renders a popover for each token. The popover is based on preact-portal, it's short so I make some modifications and copy that into index.html.

Dark mode:

Light mode:

For bytes, I just add a bottom border line to split them: (https://github.com/ggerganov/llama.cpp/assets/3001525/ad92444e-58cc-445a-b8a9-44704236e285)

(Screenshots updated after 04b6f2c)

We can set More options -> Show Probabilities to use n_probs param.

[ghost]

Ah, this is fun. It works as expected on Android too.

[jubruckne]

Very nice! But maybe you could round the probabilities to 2 decimals or so?

[jhen0409]

Very nice! But maybe you could round the probabilities to 2 decimals or so?

Sure, I'll do it later. Thanks! Maybe just change the numbers to percentage.

[staviq]

I just had the craziest idea.

I'm not requesting anything, just wondering about the viability of this idea.

Do you thing this could be used to create spelling suggestions while typing ? Like on Android keyboard ?

[SlyEcho]

If it was percentages, it would be cool.

The byte thing should be fixed in the API level, but at the time we added the probabilities, we couldn't figure it out.

[jhen0409]

The byte thing should be fixed in the API level, but at the time we added the probabilities, we couldn't figure it out.

I have read PR #1962, and I'm a bit confused about this, shouldn't we improve it by convert bytes in the UI side?

I'm thinking maybe we can do the merge for bytes to get a readable result (also helpful for Chinese or other language users), but I'm not sure if it will have other problems.

[jhen0409]

I'm thinking maybe we can do the merge for bytes to get a readable result (also helpful for Chinese or other language users), but I'm not sure if it will have other problems.

I've confirmed the another bytes pair is not able to decode successfully, so I just hide that. I see that Open AI playground also doing the same thing.

[SlyEcho]

There is some strange thing that happens sometimes: the probabilities disappear from the result.

[jhen0409]

There is some strange thing that happens sometimes: the probabilities disappear from the result.

[image]

Not very sure why it happened, maybe the completion_probabilities of some partial responses is not an array, but as I know in the server.cpp, it should have ensured that it is an array.

I just removed the array check of completion_probabilities for messages, only check params.n_probs > 0 for that, it should be avoid this problem.

[ghost]

Still working as expected on Android! ♥️

Edit: Using Iceraven(Firefox)

[jhen0409]

I also made the last to_send have a partial response, so we can correctly get the probabilities of last message. (the final response included all probabilities).

[ghost]

The problem should be fixed now, I was mistake for completion_probabilities usage, so previously it cannot show probabilities for this partial response:

Thank you for the fixes, my testing shows it's working.
For my own understanding, when you refer to a "partial response", do you mean how the tokenizer peices words together? i.e. with llama!, the tokenizer chops it in parts, is that what's meany by partial response?

Thank you.

[slaren]

I tested the same generation again, but now there is a missing token (ll):

The probs are still there, but hidden in the space:

[jhen0409]

The problem should be fixed now, I was mistake for completion_probabilities usage, so previously it cannot show probabilities for this partial response:

Thank you for the fixes, my testing shows it's working. For my own understanding, when you refer to a "partial response", do you mean how the tokenizer peices words together? i.e. with llama!, the tokenizer chops it in parts, is that what's meany by partial response?

Thank you.

[screenshot]

Generally, the partial response is included a single llama_eval return value and send via event stream, so it will be an one token. Multibytes (like emoji) is also processed in server.cpp, so it will include 2~4 tokens in a partial response.

But things like llama probably shouldn't be a partial response, and I think I found the reason, you can see my next comment.

[jhen0409]

I tested the same generation again, but now there is a missing token (ll): [image]

The probs are still there, but hidden in the space: [image]

Confirmed it was a problem from find_partial_stop_string in server.cpp, tokens like L or l are incorrectly considered as a partial stop word.

Log

token_text:  L
pos: 11
stop_pos: 18446744073709551615
final stop_pos: 1 (wrong)
to_send:  (empty)

token_text: l
pos: 12
stop_pos: 18446744073709551615
final stop_pos: 0 (wrong)
to_send: (empty)

token_text: ama
pos: 12
stop_pos: 18446744073709551615
final stop_pos: 0 (wrong)
to_send: (empty)

I'll fix this later.

UPDATED: The fix is here but it was problem with sent_token_probs_index, the above log is expected as we need to wait for possible stop words.

[jhen0409]

Before this fix, the to_send is a whitespace when it gets stop_pos 1 from L, then the sent_token_probs_index will be incorrect.

[jhen0409]

Also, I merged the master branch, so need to use GGUF models for testing here.

[slaren]

I noticed that each interaction has increasingly more vertical space. Also happens without token probabilities.

[jhen0409]

I noticed that each interaction has increasingly more vertical space. Also happens without token probabilities.

[image]

I got the same in master, in this case the model responds content like \n\nUser: and we cut the antiprompt User:. It should be easy to improve.

[jhen0409]

After fixing the newline issue, I think this can be merged. Thank you guys!

[IgnacioFDM]

Nice one. What I'd like to have in the future is a notebook mode (so basic completion instead of chat). Do you have any plans for that? I could maybe it hack it together a few weeks from now when I'm a bit less busy.

[jhen0409]

Nice one. What I'd like to have in the future is a notebook mode (so basic completion instead of chat). Do you have any plans for that? I could maybe it hack it together a few weeks from now when I'm a bit less busy.

I'm also thinking about to have a pure text completion in web UI, but the plans is not very clear. Currently I'm using the vim plugin for that, but the web UI may be could provide more visual capabilities. It's a low priority for me, but interesting.