Server UI improvements

[mounta11n]

• Make UI look nicer and cleaner overall with CSS.
• CSS outsourced as a separate file
• Added a dropdown menu with prompt style templates
• Added a dropdown menu with system prompts
• Prompt Styles and System Prompts are separate files, so editing is very easy.
• User friendly TUI: Created a script that uses "dialog" to compose the command for the server.
• Dialog Script offers the possibility to save and load configs

---

In planning or already started:

• WIP Multilingual: You will be able to select the language from a dropdown menu. So far language files only for English and German. (concerns UI elements and system prompts).
• Dark Mode
• Templates for the values of the UI options (samplers etc.), e.g. deterministic template, creative template, balanced template etc...

[FSSRepo]

Absolutely wonderful, good luck!!

[ggerganov]

Cool stuff! The dialog TUI seems very handy - will learn how it works.

Do you plan to keep updating this PR or you prefer this to get merged?

[mounta11n]

Cool stuff! The dialog TUI seems very handy - will learn how it works.

Do you plan to keep updating this PR or you prefer this to get merged?

I am glad that my work is well received!

I see that there are some formatting issues. Let me know if you want me to correct them first. Otherwise I would be happy if you merge it.

[ggerganov]

I see that there are some formatting issues. Let me know if you want me to correct them first. Otherwise I would be happy if you merge it.

Yup, try to fix these. I use editorconfig-checker locally like this:

brew install editorconfig editorconfig-checker
editorconfig-checker

[ggerganov]

Will let the PR stay open for a ~day so we gather some additional feedback. Would be great if people run some tests in the meantime to make sure it works OK

[ggerganov]

If dialog command is missing, it show just a blank screen - no error

Some more feedback:

• The start-server.sh script assumes that it is being executed from the llama.cpp/examples folder. Should be agnostic
• Stored path in config is relative - probably better to be absolute
• By default file selection starts from $HOME - better to start from llama.cpp/models/

[Azeirah]

I am not able to select and copy any generated text while still generating. Seems like a UX bug where the selection disappears with each token streamed in.

I noticed that too, it's not a new bug introduced by this PR though. I'm not sure how feasible it is to implement copy paste functionality on text that gets replaced this frequently.

I'll take a look at it though, there's always the possibility of caching the selection and just reapplying it after re-render, but I'm prettyyy sure that's going to have some nasty side-effects.

[Azeirah]

I am not able to select and copy any generated text while still generating. Seems like a UX bug where the selection disappears with each token streamed in.

I noticed that too, it's not a new bug introduced by this PR though. I'm not sure how feasible it is to implement copy paste functionality on text that gets replaced this frequently.

I'll take a look at it though, there's always the possibility of caching the selection and just reapplying it after re-render, but I'm prettyyy sure that's going to have some nasty side-effects.

Fixing it in vanilla JS turned out to be pretty simple, instead of mutating the innerText/textContent of an element, it's easier to append a Text node to the container element. This way the selection is retained because nothing is rerendered, only appended.

IE: https://codepen.io/laauurraaa/pen/MWLPJed?editors=1010

How this works in the current server UI I'm not too certain, I haven't figured out what framework it's even using. It's using a minified version of a very lightweight React-like? Possibly Preact or SolidJS? I'm not familiar with those.

[tobi]

I like the ux updates!

The chat template system is a bit conflicting with the work/thought going on at #4216 - basically, what we should do is to make the client/html dumber and have it use the openai endpoint to get chat automatically. We want the endpoint to apply the chat template that the model requires automatically.

[mounta11n]

There are no "strings attached" - if you don't have time or will to contribute in the future, that's completely fine. We don't expect any long or short term commitment and any contribution, big or small, is very welcome.

I don't have a strong preference of the style - as long as it renders OK. It does make sense to reuse the orange - yes.

Regardless of that, I'll continue working on the dark mode right away and see that I finish it as soon as I can.

No rush, have fun!

Okay thanks

I have posted some feedback/experience under https://www.reddit.com/r/LocalLLaMA/comments/18534f1/i_have_given_llamacpp_server_ui_a_facelift/

* I am not able to select and copy any generated text while still generating. Seems like a UX bug where the selection disappears with each token streamed in.

* possibly wrong suggestion, but I [got used](https://github.com/abetlen/llama-cpp-python) to have /docs endpoint with description of the endpoints available.
  It could point to/render https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md#api-endpoints at first, anyway seems helpful to have it served.

* the prob. values seem [off, not normalized](https://preview.redd.it/rmmx9kclh33c1.png?width=183&format=png&auto=webp&s=03a94c6b8d21dee12ab9822f4206752296e08172), sometimes.

Thanks for the feedback. I am not sure if I have understood you correctly. Do you mean that after you press submit/send and while the llm is generating, you are not able to select text already generated?

If so, I couldn't reproduce this error. If not, please elaborate how I could reproduce it. I will anyway investigate and see if this bug is caused by my code and if I could fix it.
I am just debugging some other UI issues, but I am almost finished with it and will start investigating right away after that.

@ALL fyi:
While I was working on to implement a dark mode, I decided to change my goal and instead of
creating a dark mode, I am creating various themes, so one could have a more personalized UI, and it is now very easy to create own color-themes.

Here is a first impression (additionally to the OpenAI-Playground-like UI):

In case someone is interested. The following output was generated by starling 7b, which indeed surprised me on how good this model is. Its understanding and producing of german language is impressive considering its size and english focused dataset.
however its coding capabilities are poor, for example it struggles with easy css or html tasks.

Fixing it in vanilla JS turned out to be pretty simple, instead of mutating the innerText/textContent of an element, it's easier to append a Text node to the container element. This way the selection is retained because nothing is rerendered, only appended.

IE: https://codepen.io/laauurraaa/pen/MWLPJed?editors=1010

Okay, that sounds good, but it is interesting that I could not reproduce this behaviour. I'll try your code out.

I like the ux updates!

The chat template system is a bit conflicting with the work/thought going on at #4216 - basically, what we should do is to make the client/html dumber and have it use the openai endpoint to get chat automatically. We want the endpoint to apply the chat template that the model requires automatically.

Oh thanks for the hint. Okay I will read what is going on at #4216 and address what you mentioned. by applying the templates automatically, do you mean to get the information from the .gguf itself? I was thinking about something like that, but found that in some models the information was missing. But anyway, I can follow your works and orient myself accordingly.

[Tostino]

@mounta11n It means that the UI wouldn't have to deal with the template at all, you just need to hit the chat/completions endpoint with a list of messages with a role:content pairs, and the server will handle templating it out and returning just the model's response.

For most models, they seem to stick to the system, assistant, and user roles...but that is far from guaranteed.
For example, my model has a few extra roles that hydrate some extra variables in the template if they are passed in (meta-current_date, meta-task_name), and then I also have an additional role for any content that the user is asking about (user_context) that can be interleaved with the user and assistant messages.

e.g.

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer " \
  -d '{
    "model": "Tostino/Inkbot-13B-8k-0.2",
    "stop": ["\n<#user#>", "\n<#bot#>"],
    "stream": false,
    "add_generation_prompt": true,
    "echo": false,
    "n": 1,
    "messages": [
	{"role": "meta-current_date", "content": "2023-10-20"},
	{"role": "meta-task_name", "content": "general"},
	{"role": "system", "content": "You are a helpful assistant."},
	{"role": "user", "content": "What color is the car?"},
	{"role": "user_context", "content": "I was driving around last Saturday in my Blue Honda Accord."}
	]
  }'

I'm not saying "implement support for my use case" here, I just wanted to bring it up so you are aware of what is possible.

You can see how the endpoint for vLLM that I implemented works, fingers crossed when llama.cpp implements it, they try and stay compatible with the choices that were made.

[mounta11n]

Here is some little update. the dark mode/light mode work is finished and here are the various themes:

Light Themes

Default and Boring, but could maybe a good starting point for customizing own themes

The already shown OAI-Playground-like theme

Looks professional

Dark Themes

Mango Tango - inspired by llama.cpp logo/banner

Nord Theme

favorite

---

As already mentioned, the work on the UI elements and themes has been completed. Now it's on to the next work: figuring out how to fix the misbehaviors mentioned by other users (plus I've discovered a few more bugs in the meantime). This mainly occurs when toggling to completion and then back to chat. I would appreciate any support, especially if someone who is very familiar with Javascript and has already done a " eye diagnosis" could give me some advice.

[Azeirah]

figuring out how to fix the misbehaviors mentioned by other users (plus I've discovered a few more bugs in the meantime). This mainly occurs when toggling to completion and then back to chat. I would appreciate any support, especially if someone who is very familiar with Javascript and has already done a " eye diagnosis" could give me some advice.

I can pick up the JS issues. I think it's best to merge this as-is, you haven't introduced any JS bugs as far as I know?

We can open a new issue and list the bugs you're aware of. I'm a web dev and I can look at the JS parts and maybe others can look at it too.

That way your styles become available for everyone to use, and you don't have to worry about the JS bugs and they'll be fixed soon regardless!

[mounta11n]

I can pick up the JS issues. I think it's best to merge this as-is, you haven't introduced any JS bugs as far as I know?

We can open a new issue and list the bugs you're aware of. I'm a web dev and I can look at the JS parts and maybe others can look at it too.

That way your styles become available for everyone to use, and you don't have to worry about the JS bugs and they'll be fixed soon regardless!

Oh that would be very helpful indeed! I would be very pleased about that. Do I have to take any specific action now?

[Azeirah]

I can pick up the JS issues. I think it's best to merge this as-is, you haven't introduced any JS bugs as far as I know?
We can open a new issue and list the bugs you're aware of. I'm a web dev and I can look at the JS parts and maybe others can look at it too.
That way your styles become available for everyone to use, and you don't have to worry about the JS bugs and they'll be fixed soon regardless!

Oh that would be very helpful indeed! I would be very pleased about that. Do I have to take any specific action now?

I suppose wait for @ggerganov for feedback/merging. Other than that, it's helpful if you could open an issue with a list of the bugs present in the UI with steps on how to reproduce the bugs

[ggerganov]

• When I toggle between "Chat" and "Complete", the style control stops working
• I was hoping that the "Default" style would remain as the one on master

Regarding templates - probably this PR would need to be split into UI changes and template changes. The goal is to have the server apply the correct chat template automatically, so the template / prompt functionality on the client would eventually get deprecated.

[mounta11n]

I can pick up the JS issues. I think it's best to merge this as-is, you haven't introduced any JS bugs as far as I know?
We can open a new issue and list the bugs you're aware of. I'm a web dev and I can look at the JS parts and maybe others can look at it too.
That way your styles become available for everyone to use, and you don't have to worry about the JS bugs and they'll be fixed soon regardless!

Oh that would be very helpful indeed! I would be very pleased about that. Do I have to take any specific action now?

I suppose wait for @ggerganov for feedback/merging. Other than that, it's helpful if you could open an issue with a list of the bugs present in the UI with steps on how to reproduce the bugs

Just to be sure if I understand you right and not gonna doing something dumb: Do you mean open an issue here in gerganov's repo, even this pr is not merged yet? I am asking because for now I/we do not know if the at least some of the bugs are the result of my code. So I would open an issue with the bugs I know and as a sidenote refer to this pr, right?

PS: one bug that I noticed is described by ggerganov, seens to occur there too.

• When I toggle between "Chat" and "Complete", the style control stops working

• I was hoping that the "Default" style would remain as the one on master

Regarding templates - probably this PR would need to be split into UI changes and template changes. The goal is to have the server apply the correct chat template automatically, so the template / prompt functionality on the client would eventually get deprecated.

I am sorry, I didn't understand your second point. What do you mean by "Default" style? Do you mean that one I introduced in my last post and which I described as 'boring'? Or do you mean the original style, the one before I made this pr at all? Or the oai-playground-like one, the one I have on my fork/master?

Yes, I have heard that there are discussions about integrating the prompt templates into the program logic and standardizing the prompt interface. We can do it the way you suggest if you think that's better.

My thought on this is that it would make sense to keep it here in this pull request after all, because this is a long awaited feature (e.g. the resonance on this pr in reddit LocalLlama was very positive and people are waiting for the merge there, in large part because the prompt style templates are expected).

The other point is that a second pr will probably not be necessary if a different and more elegant solution is already being worked on anyway. So my suggestion would be that the prompt style templates remain temporary to bridge the gap until the other project is ready. There could be a note on the server README.md (and perhaps also in the UI itself) that the current implementation of the prompt styles will very probably soon be obsolete.

Let me know how I should proceed : )

Aside from that, just want to double check: Regarding the system prompts, do we agree that they should stay and that they fulfill a useful function in the frontend?

ps: in the meantime i have continued to work on the UI and replaced the dropdown menu for the color styles with a nicer css menu and also changed the layout slightly. e.g. the color styles menu has moved up two 'sections' because it has a very superordinate role. Ironically, as a result, the bug where the styles could no longer be changed after toggling between chat and completion is no longer reproducible.

[ggerganov]

Just to be sure if I understand you right and not gonna doing something dumb: Do you mean open an issue here in gerganov's repo, even this pr is not merged yet? I am asking because for now I/we do not know if the at least some of the bugs are the result of my code. So I would open an issue with the bugs I know and as a sidenote refer to this pr, right?

The one that is on the master branch, before this PR.

Aside from that, just want to double check: Regarding the system prompts, do we agree that they should stay and that they fulfill a useful function in the frontend?

I think so. From what I gather, the system prompt can be used with some of the chat templates.

Yes, I have heard that there are discussions about integrating the prompt templates into the program logic and standardizing the prompt interface. We can do it the way you suggest if you think that's better.

My thought on this is that it would make sense to keep it here in this pull request after all, because this is a long awaited feature (e.g. the resonance on this pr in reddit LocalLlama was very positive and people are waiting for the merge there, in large part because the prompt style templates are expected).

I don't have a strong opinion on this. If the frontend logic for supporting the templates is lightweight enough and can be easily deprecated in the future, then I guess it is fine.

[mounta11n]

@ggerganov sorry for the delayed response. In the meantime I've decided to split this PR into two. One for the server improvements and another with the dialog-start-server script.

The current state of things is that I have left the old UI as the default one and as index.html allmost untouched, beside that I've added a linking button to the new UI as you can see in the image. The same is with the new UI, which is now index-new.html and it has a link to the old UI.

Please tell me if there is something else to do in the front end. I will try to hurry up, cause I pretty much hope to get the new ui merged before christmas : D

[mounta11n]

And oh, I don't know what kind of mess I made with the files shown as changed. Actually it should only affect examples/server/README.md and files within examples/server/public.

Is it possible to fix or undo this?

[AmgadHasan]

Hey @mounta11n
Thanks for working on this. The new UI looks really good. Any updates about the PR?.
Edit:
#7633
I guess we can close this now.

[ggerganov]

A new UI has already been merged via #7633