Description
While troubleshooting #2678, I caught llama2 (or something?) generating a bunch of non-printable characters (ASCII 28 specifically) in my bot conversations. It is possible that this is the root cause of #2678
Since I was feeding LLM responses into a transcript in the context, it seems then these pollute the conversation (server.cpp does not directly write responses into the context like how main.cpp does). This would cause my next round of token processing to go crazy and spin the GPU but not return anything (presumably trying to do something w the control characters)
I was using upstage-llama-2-70b-instruct-v2.ggmlv3.q5_K_M.bin on Metal M1 64GB. I am not using MPS
This was observed with build bf83bff
I have not been able to reliably repro this yet with dadbed9 so take it with a grain of salt, but if anyone else is suffering from #2678 check to see if the data returned contains non-printables. I'm not sure if these are coming from code or data but how those got the 70b data in the first place is probably another question
Example here
#2678 (comment)
Activity
[-]llama2 sometimes generates non-printables such as ASCII 28 (\xc1) which can contaminate context for subsequent processing[/-][+]llama sometimes generates non-printables such as ASCII 28 (\xc1) which can contaminate context for subsequent processing[/+][-]llama sometimes generates non-printables such as ASCII 28 (\xc1) which can contaminate context for subsequent processing[/-][+]llama sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context[/+][-]llama sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context[/-][+]Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context[/+][-]Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context[/-][+]Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context / cause havoc[/+]lshzh-ww commentedon Aug 24, 2023
Hi @ProjectAtlantis-dev ,
Could you attempt to backport #2699 to your code? I assume you are not using the latest llama.cpp code and are missing the correct fix for the problem you reported.
ProjectAtlantis-dev commentedon Aug 24, 2023
I am testing with dadbed9 and have not gotten any more control characters.
In any case, it is a pre-GGUF model so perhaps has been overcome by events
ProjectAtlantis-dev commentedon Aug 29, 2023
Have not seen any more control chars and since pre-GGUF model I am closing