AdvisedRequest removed with RC1

[uwilken]

Hi,

I am working on a project where we share data between Tools and RequestAdvisors.

Tools add data to the tool context and an Advisor uses this data to add additional annotations to the final answer.

With #2993 the class AdvisedRequest has been removed. Instead, ChatClientRequests are used now in Advisors.

In ChatClientRequest it is possible to access the context that is available to all other Advisors, but there is no way to access the tool context (formerly AdvisedRequest#toolContext).

@markpollack, @ThomasVitale, Is there a way to access the tool context in an Advisor?

Your feedback is much appreciated!
Ulrich

[ThomasVitale]

@uwilken the ChatClientRequest gives you access to the fully formed Prompt now, from which you can retrieve the tool context.

ChatOptions chatOptions = chatClientRequest.prompt().getOptions();
if (chatOptions instanceof ToolCallingChatOptions toolCallingChatOptions) {
	Map<String, Object> toolContext = toolCallingChatOptions.getToolContext();
}

I hope that helps!

[uwilken]

Thank you very much @ThomasVitale !

I was able to retrieve the tool context from the ChatClientRequest as you proposed. 👍

Unfortunately I just retrieved the context data that was added when the chat options have been initialized via ChatClientRequestSpec#toolContext(Map<String, Object> toolContext).

What we are trying to extend the ToolContext with data that gathered during tool execution. Later on in the Advisor I want to retrieve the data again.

Our tools implement the BiFunction IF and via java.util.function.BiFunction#apply we have access to the ToolContext that we extend on the fly:

public ToolResponse<R> apply(T t, ToolContext toolContext) {
...
toolContext.getContext().put("foo", "bar");
...}

But unfortunately I cannot find the data afterwards in StreamAdvisor#adviseStream(ChatClientRequest chatClientRequest, StreamAdvisorChain streamAdvisorChain) when trying to read them via chatClientRequest.prompt().getOptions().getToolContext().

I can just find the data that was added via ChatClientRequestSpec#toolContext(Map<String, Object> toolContext) initially.

Am I doing something wrong?

[ThomasVitale]

@uwilken If I understood correctly, you add new items to the ToolContext from within a tool, and then you want to read them back from an Advisor. Did I understand that correctly?

Did you have this use case working correctly with the previous AdvisedRequest API? I'm asking because, if I'm not wrong, the ToolContext has always been a uni-directional channel to pass context from a chat model/client to the tool execution (without the possibility to go the other way around).

[uwilken]

@ThomasVitale Yes, with AdvisedRequest#toolConext we can access data that has been added to the ToolContext during tool execution. This works with version 1.0.0-M8.

If the ToolContext is not meant to work this way around, is there another way to collect data during tool execution and pass it on so it can be consumed later on in com.coremedia.ai.kio.backend.advisors.RequestAdvisor#after?

[tzolov]

@uwilken, AFAIK, ToolContext wasn't designed to return data from tools to the application, though we didn't block this functionality.
I understand how this can be an important use case (especially since SpringAI controls tool execution by default).
I can't think of workaround currently, but we'll address this regression of an unintentional feature in version 1.0.1.

[uwilken]

Hi @tzolov,
thank you for your feedback. This function is crucial for our application and I could imagine it being useful for others as well. I will stick with 1.0.0-M8 for now and hope that 1.0.1 will support it again.
Do you have a rough time estimate for 1.0.1?

[uwilken]

@tzolov, do you know already if it will be supported with v1.0.1 again to pass context data from tools back to the application?