Prompt for instruction models to be used in server

[lucasjinreal]

Hi, currently using the API, i can not get a complete reply from client side,

the sentence are stopped at unwanted token, but with cli, it is normal. Why? How to specific prompt template?

[zhiyixu]

I probably encountered the same problem in #384

[lucasjinreal]

@zhiyixu @gjmulder Yes. my situation a little difference, the hyper params would effect, but my model even worse, I am using a totally different promp template (vicuna like), How can i set it in server? can not found anywhere to set it.

[lucasjinreal]

@gjmulder As I said, i didn't found prompt template....... did you saw it?

didn't know which one should map to your params as well.....
--temp 0.7 --top_k 40 --top_p 0.5 --repeat_last_n 256 --repeat_penalty 1.17647

[gjmulder]

Sorry @lucasjinreal, I should have pasted the server help to @zhiyixu 's issue thread.