Adds LanguageModelFailurePlugin. Closes #1313

[waldekmastykarz]

Adds LanguageModelFailurePlugin. Closes #1313

To test:

• enable the plugin

{
  "$schema": "https://github.com/raw/dotnet/dev-proxy/main/schemas/v1.0.0/rc.schema.json",
  "plugins": [
    {
      "name": "LanguageModelFailurePlugin",
      "enabled": true,
      "pluginPath": "~appFolder/plugins/DevProxy.Plugins.dll"
    }
  ],
  "urlsToWatch": [
    "*"
  ],
  "languageModel": {
    "enabled": true
  },
  "asSystemProxy": false
}

• call Ollama using its proprietary API

curl -ikx http://127.0.0.1:8000 -X POST http://localhost:11434/api/generate -H "Content-Type: application/json" -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

• call Ollama using its OpenAI-compatible chat completion API

curl -ikx http://127.0.0.1:8000 -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [
      {
        "role": "user",
        "content": "Why is the sky blue?"
      }
    ]
  }'

By default, the plugin applies one of its failures at random. You can specify one or more failures you want to test specifically through plugin's configuration:

{
  "$schema": "https://github.com/raw/dotnet/dev-proxy/main/schemas/v1.0.0/rc.schema.json",
  "plugins": [
    {
      "name": "LanguageModelFailurePlugin",
      "enabled": true,
      "pluginPath": "~appFolder/plugins/DevProxy.Plugins.dll",
      "configSection": "languageModelFailurePlugin"
    }
  ],
  "urlsToWatch": [
    "*"
  ],
  "languageModel": {
    "enabled": true
  },
  "languageModelFailurePlugin": {
    "failures": [
      "Overgeneralization",
      "PlausibleIncorrect"
    ]
  },
  "asSystemProxy": false
}

Available failure types by default:

• AmbiguityVagueness
• BiasStereotyping
• CircularReasoning
• ContradictoryInformation
• FailureDisclaimHedge
• FailureFollowInstructions
• Hallucination
• IncorrectFormatStyle
• Misinterpretation
• OutdatedInformation
• OverSpecification
• OverconfidenceUncertainty
• Overgeneralization
• OverreliancePriorConversation
• PlausibleIncorrect

You can also add your own by adding a new .prompty file in the ~appFolder/prompts file. The file must be named lmfailure_<failure>.prompt. failure must be written kebab case (my-failure) and in plugin config you can refer to it PascalCase (MyFailure).

This PR also replaces the Prompty.Core SDK with a custom parser, which is easier and easier to use for our needs.

[garrytrinder]

I'm getting a fail log when starting Dev Proxy.

Ollama is running in the background and the requests are succesful but being passed through.

I'm using the latest Ollama build, v0.9.6

[waldekmastykarz]

Can you run Dev Proxy with trace logging to see what's wrong?

[waldekmastykarz]

Ah, I just saw that I missed enabling language model in my sample config. Sorry 🤦‍♂️