MessageChatMemoryAdvisor results in an error if prompt is initialized with List<Message> instead of userText

[rwankar]

Bug description
Create a prompt with

Prompt prompt = new Prompt(messages); // list of messages including UserMessage

And set MessageChatMemoryAdvisor as an advisor.

chatClient
  .prompt(prompt)
  .system(systemText)
  .advisors(new MessageChatMemoryAdvisor(chatMemory))

Results in an error that Content cannot be null because MessageChatMemoryAdvisor is looking for userText() which is not present but the prompt already has the UserMessage in list of messages.

Environment
Spring AI 1.0.0-SNAPSHOT
SpringBoot 3.4.2
Java 21

[dev-jonghoonpark]

What chat model and chat memory are you using?
Can you attach an error messages?

[rwankar]

I'm using Azure OpenAI

Here is the stack trace..

java.lang.IllegalArgumentException: Content must not be null for SYSTEM or USER messages
    at org.springframework.util.Assert.notNull(Assert.java:181) ~[spring-core-6.2.2.jar!/:6.2.2]
    at org.springframework.ai.chat.messages.AbstractMessage.<init>(AbstractMessage.java:69) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.messages.UserMessage.<init>(UserMessage.java:62) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.messages.UserMessage.<init>(UserMessage.java:49) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor.before(MessageChatMemoryAdvisor.java:97) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor.aroundCall(MessageChatMemoryAdvisor.java:62) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundCall$1(DefaultAroundAdvisorChain.java:98) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at io.micrometer.observation.Observation.observe(Observation.java:564) ~[micrometer-observation-1.14.3.jar!/:1.14.3]
    at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundCall(DefaultAroundAdvisorChain.java:98) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at com.celoxis.psa.base.ai.HintChatMemoryAdvisor.aroundCall(HintChatMemoryAdvisor.java:47) ~[!/:14.5.0] 
    at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.lambda$nextAroundCall$1(DefaultAroundAdvisorChain.java:98) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at io.micrometer.observation.Observation.observe(Observation.java:564) ~[micrometer-observation-1.14.3.jar!/:1.14.3] 
    at org.springframework.ai.chat.client.advisor.DefaultAroundAdvisorChain.nextAroundCall(DefaultAroundAdvisorChain.java:98) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.doGetChatResponse(DefaultChatClient.java:493) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT] 
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.lambda$doGetObservableChatResponse$1(DefaultChatClient.java:482) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at io.micrometer.observation.Observation.observe(Observation.java:564) ~[micrometer-observation-1.14.3.jar!/:1.14.3]
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.doGetObservableChatResponse(DefaultChatClient.java:482) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.doGetChatResponse(DefaultChatClient.java:466) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]
    at org.springframework.ai.chat.client.DefaultChatClient$DefaultCallResponseSpec.chatResponse(DefaultChatClient.java:510) ~[spring-ai-core-1.0.0-SNAPSHOT.jar!/:1.0.0-SNAPSHOT]

[dev-jonghoonpark]

ChatMemory chatMemory = new InMemoryChatMemory();
String content = ChatClient.builder(chatModel).build()
        .prompt("hi")
        .system("systemText")
        .advisors(new MessageChatMemoryAdvisor(chatMemory))
        .call()
        .content();
System.out.println(content);

I have tried testing in the same environment, but it is not reproduced.

Could you check if the prompt or systemText variable is null?
The logs show the message: "Content must not be null for SYSTEM or USER messages."

[rwankar]

If you create prompt from text it works. You need to create Prompt from a list of messages which contains a UserMessage. I've mentioned this in my first post.

The stack trace show an issue on line 97 in MessageChatMemoryAdvisor

UserMessage userMessage = new UserMessage(request.userText(), request.media());

It should probably create a UserMessage only if request.userText() is not null.

[dev-jonghoonpark]

Prompt prompt = new Prompt(List.of(new UserMessage("hi")));

ChatMemory chatMemory = new InMemoryChatMemory();
String content = ChatClient.builder(chatModel).build()
        .prompt(prompt)
        .system("systemText")
        .advisors(new MessageChatMemoryAdvisor(chatMemory))
        .call()
        .content();
System.out.println(content);

This also works. It would be nice if you could provide a more specific example.

[rwankar]

Here is a working example. It's a SpringBoot maven project. The example is a bit about getting the weather in a city. There are two tools provided. getZip() and getWeather(). To get the weather in a city, the LLM should first call the getZip tool to get the zip code and then use the zip code to call the getWeather() function.

I'm trying to get the ChatMemory updated with all the messages including the tool call and tool call result. So I'm following the example as suggested in the "User-controlled tool execution" in the Tools documentation.

There are 4 files.
AiClient.java - responsible for building the ChatClient with the tools and the actual Chat.
Application.java - The main SpringBoot app
ChatController.java - The RestController with /chat GET endpoint.
ApiKeys.java - reads the keys from Environment.

Set 2 environment variables
azure.openai.key
azure.openai.endpoint

Access localhost:8080/chat?q=what is the weather in Seattle?

On a side note, ToolCallingChatOptions.builder().toolCallbacks() expects List<FunctionCallback>. But FunctionCallback is deprecated. Should I be doing this a different way?

spring-ai-test.zip

[rwankar]

Were you able to reproduce the issue?

[markpollack]

There has been a big redesign of the advisor chain specifically to address these kind of issues.

There is now a test that aim to reproduce what was described here in a test and it does not throw an exception. See commit 10ff11d

If there are any issues, please reopen.