Feature request: add support for streaming tool use

[lsorber]

The combination stream=True, tool_choice="auto" raises an exception right now, which means that developers are stuck with one of two unfortunate choices:

1. Developing an application that streams the response but cannot use tools
2. Developing an LLM application that can use tools but cannot stream the response

Relevant discussion: #1615

[SaymV]

Admittedly this is the wrong place to ask this question but as a beginner I feel like you're the right person to answer:

Does something need to be done to llama.cpp directly in order to handle streaming tool calling? I see from your feature branch that you added a RAG layer to this python implementation. I ask because I built llamma.cpp from source figuring it would be better optimized for my system, but I am stuck with this server error
{"code":500,"message":"Cannot use tools with stream","type":"server_error"}.

Is it the case that if I installed the pre-built python version that this would go away?

Edit: I see here that there's a PR in draft. We're too close to the bleeding edge!

[edmcman]

Llama.cpp is still waiting on ggml-org/llama.cpp#12379

I'm not sure how this python library handles tools. I think it is somewhat different though.

[kooshi]

ggml-org/llama.cpp#12379
has been merged!