Description
I have a question about the id
field in the data returned from the completions
endpoint. I see that there's a unique ID that identifies what completion a message is part of, and I'm wondering if this is only data for the client, or whether it has additional functionality.
Eventually I'm hoping to have a a couple different models running on my server and I'm trying to figure out if there's a mechanism that exists for a sort of chat functionality with unique contexts. Llama.cpp recently gained the ability to run multiple instances at once without much overhead, so I'm looking for a way to keep a unique context between a couple conversation 'threads'.
Is there any mechanism, or is there a plan for one? Just want to make sure I'm not missing something if it's built already xD
{
"id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"object": "text_completion",
"created": 1679561337,
"model": "models/7B/...",
"choices": [
{
"text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
"index": 0,
"logprobs": None,
"finish_reason": "stop"
}
],
...
}
Activity
abetlen commentedon Apr 3, 2023
It's currently just for the client and not saved anywhere.
We could include some kind of instance id as part of the unique id scheme, I don't think it would effect interoperability with services that expect OpenAI response objects as the id scheme isn't part of their public API (AFAIK).
As for chat mode and continuing completion, it looks like this is on the horizon for
llama.cpp
but is still not completely supported by the api.This is something I'm following so hopefully once the full api for managing state is complete we can integrate chat sessions.