Being more lax with API responses to allow for better compatibility with alternative services?

[spantaleev]

There are various services offering an OpenAI-compatible API nowadays. Some are doing things a bit differently, which makes async-openai choke on their responses.

I've managed to reproduce the problem on TogetherAI (frequently down though) and OpenRouter. The latter seems more reliable and possibly provides some models that can be used for free, so it's probably easier to test against.

On successful chat completion generation, OpenRouter returns an API response that looks like this:

{
  "id": "gen-abcd",
  "model": "model-here",
  "object": "chat.completion",
  "created": 1725709829,
  "choices": [
    {
      "logprobs": null,
      "finish_reason": "eos",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello there! How can I help you today? If you have any questions, I'm here to assist.",
        "refusal": ""
      }
    }
  ],
  "usage": {
    "prompt_tokens": 26,
    "completion_tokens": 24,
    "total_tokens": 50
  }
}

.. which async-openai chokes on with:

failed to deserialize api response: unknown variant eos, expected one of stop, length, tool_calls, content_filter, function_call at line 51 column 180

---

Since I'm building a tool which aims to support both OpenAI and other OpenAI-compatible API endpoints, I either need to use both async-openai and another (more lax) library.. Or I need to open issues for all OpenAI-compatible providers and have them try to mimic OpenAI's API responses better (e.g. avoid using finish_reason = eos). Here's an instance where some folks fixed their API-compatibility to satisfy async-openai, but.. persuading various providers to do the same seems like a difficult endeavour.

It would probably be easier if async-openai was not so strict sometimes. This possibly goes in other directions as well - not taking pre-defined enums for certain parameters and letting people use strings instead (perhaps via an SomeEnum::Custom(String) variant).

I'm not sure if it's the goal of this library to allow working with OpenAI-compatible APIs or if it's something that you're against.

[64bit]

Thanks for the issue, sorry to hear that others are selling you "compatible" api when they are not, please see https://github.com/64bit/async-openai?tab=readme-ov-file#contributing

[spantaleev]

Oh, I'm sorry for having missed this section of the README. It definitely clears things up - async-openai adheres to whatever the OpenAI spec is.

Perhaps having this issue here for posterity will help others figure it out quicker. Thank you for settling this quickly!

[64bit]

Please consider using Bring your own types feature.