Handle additional usage details in Anthropic responses #1549
Conversation
|
@DouweM it looks like the p39 test suite failed due to something unrelated to this PR (a network error when downloading a dependency). I'm not sure if it's just flakiness, but perhaps someone with the access could rerun the build? |
| if isinstance(value, int): | ||
| details[key] = value | ||
|
|
||
| # Usage coming from the RawMessageDeltaEvent doesn't have input token data, hence these getattr calls | ||
| request_tokens = getattr(response_usage, 'input_tokens', None) |
There was a problem hiding this comment.
Can we simplify this by defaulting to 0 here and below, and skipping the isinstance calls?
There was a problem hiding this comment.
We can, and that will definitely be simpler, but it might change behavior elsewhere. I did it this way specifically since usage.Usage.request_tokens supports None, and so do the cache_* fields on the Anthropic Usage object and wanted to represent them faithfully. If I coerce request_tokens to 0, the usage.Usage object will tell callers that there is a count of 0 tokens rather than that attribute being missing (such as when processing a RawMessageDeltaEvent event).
That said, more than happy to make this change, just want to make sure this was considered
This change handles the two additional usage fields returned by Anthropic:
cache_creation_input_tokensandcache_read_input_tokens. When prompt caching is not enabled, this change has no impact.Currently, the providers/models system is very flexible, which has made it easy to enable Anthropic's prompt caching by providing a custom client or provider, but this part of the code
_map_usageis not nicely overridable, so updating it to handle the full usage object properly.This, or a similar, change, will be need to support prompt caching natively too, once someone gets to that.