Skip to content

Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context / cause havoc #2758

Closed
@ProjectAtlantis-dev

Description

@ProjectAtlantis-dev

While troubleshooting #2678, I caught llama2 (or something?) generating a bunch of non-printable characters (ASCII 28 specifically) in my bot conversations. It is possible that this is the root cause of #2678

Since I was feeding LLM responses into a transcript in the context, it seems then these pollute the conversation (server.cpp does not directly write responses into the context like how main.cpp does). This would cause my next round of token processing to go crazy and spin the GPU but not return anything (presumably trying to do something w the control characters)

I was using upstage-llama-2-70b-instruct-v2.ggmlv3.q5_K_M.bin on Metal M1 64GB. I am not using MPS

This was observed with build bf83bff

I have not been able to reliably repro this yet with dadbed9 so take it with a grain of salt, but if anyone else is suffering from #2678 check to see if the data returned contains non-printables. I'm not sure if these are coming from code or data but how those got the 70b data in the first place is probably another question

Example here
#2678 (comment)

Activity

changed the title [-]llama2 sometimes generates non-printables such as ASCII 28 (\xc1) which can contaminate context for subsequent processing[/-] [+]llama sometimes generates non-printables such as ASCII 28 (\xc1) which can contaminate context for subsequent processing[/+] on Aug 24, 2023
changed the title [-]llama sometimes generates non-printables such as ASCII 28 (\xc1) which can contaminate context for subsequent processing[/-] [+]llama sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context[/+] on Aug 24, 2023
changed the title [-]llama sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context[/-] [+]Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context[/+] on Aug 24, 2023
changed the title [-]Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context[/-] [+]Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context / cause havoc[/+] on Aug 24, 2023
lshzh-ww

lshzh-ww commented on Aug 24, 2023

@lshzh-ww
Contributor

Hi @ProjectAtlantis-dev ,

Could you attempt to backport #2699 to your code? I assume you are not using the latest llama.cpp code and are missing the correct fix for the problem you reported.

ProjectAtlantis-dev

ProjectAtlantis-dev commented on Aug 24, 2023

@ProjectAtlantis-dev
Author

I am testing with dadbed9 and have not gotten any more control characters.

In any case, it is a pre-GGUF model so perhaps has been overcome by events

ProjectAtlantis-dev

ProjectAtlantis-dev commented on Aug 29, 2023

@ProjectAtlantis-dev
Author

Have not seen any more control chars and since pre-GGUF model I am closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @ProjectAtlantis-dev@lshzh-ww

        Issue actions

          Inference sometimes gens control chars such as ASCII 28 (\xc1) which can pollute context / cause havoc · Issue #2758 · ggml-org/llama.cpp