Response API Error 500 when telemetry enabled and using gemini models

[mhdawson]

System Info

• LlamaStack Version: 0.2.18 (distribution-starter)

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

Summary

LlamaStack's responses API fails with 'ModelResponseStream' object has no attribute 'usage' when using Gemini models,
preventing the API from functioning properly.

Environment

• LlamaStack Version: 0.2.18 (distribution-starter)
• Affected Models: All Gemini models (tested with gemini-2.0-flash, gemini-2.5-flash, gemini-2.5-pro)
• Provider: remote::vertexai
• API Endpoint: /v1/openai/v1/responses
• Deployment: Kubernetes/OpenShift

Steps to Reproduce

1. Configure LlamaStack with a Gemini model using the vertexai provider
2. Enable telemetry (default configuration)
3. Make a request to the responses API:

import openai
client = openai.OpenAI(base_url="http://llamastack:8321/v1/openai/v1")
response = client.responses.create(
    input=[{"role": "user", "content": "Hello", "type": "message"}],
    model="appeng-ai-quickstarts-vertexai/vertex_ai/gemini-2.0-flash",
    stream=False
)

Expected Behavior

• Responses API should return a successful response object
• Telemetry should handle missing usage attributes gracefully

Actual Behavior

• HTTP 500 Internal Server Error
• Server logs show: 'ModelResponseStream' object has no attribute 'usage'
• API request fails completely

Root Cause Analysis

The issue occurs in llama_stack/core/routers/inference.py in the openai_chat_completion method (lines ~532-536):

if self.telemetry:
    metrics = self._construct_metrics(
        prompt_tokens=response.usage.prompt_tokens,      # ← FAILS HERE
        completion_tokens=response.usage.completion_tokens,
        total_tokens=response.usage.total_tokens,
        model=model_obj,
    )

Problem: The code unconditionally accesses response.usage attributes for telemetry logging, but Gemini's ModelResponseStream objects do not provide a usage attribute.

Additional locations with same issue:

• Line ~428-430 in openai_completion method
• Line ~801-805 in streaming section

Impact

• Severity: High - Completely blocks responses API usage with Gemini models
• Scope: Affects all Gemini model variants when telemetry is enabled
• Workaround: Disable telemetry entirely (removes observability)

Proposed Fix

Add defensive checks before accessing usage attributes:

if self.telemetry and hasattr(response, 'usage') and response.usage is not None:
    metrics = self._construct_metrics(
        prompt_tokens=response.usage.prompt_tokens,
        completion_tokens=response.usage.completion_tokens,
        total_tokens=response.usage.total_tokens,
        model=model_obj,
    )
    # ... rest of telemetry logic

Apply this pattern to all locations that access response.usage or chunk.usage.

Additional Context

• OpenAI chat completions API works fine with the same Gemini models
• Issue is specific to responses API internal implementation
• Both LlamaStack and OpenAI clients hit the same server-side error
• Telemetry disabling workaround confirmed - removing telemetry from config resolves the issue

Test Case

# This should work without throwing AttributeError
response = client.responses.create(
    input=[{"role": "user", "content": "test", "type": "message"}],
    model="appeng-ai-quickstarts-vertexai/vertex_ai/gemini-2.0-flash"
)
assert response.status == "completed"

Files to Modify

• llama_stack/core/routers/inference.py (primary fix location)
• Any other files that unconditionally access .usage attributes

Error logs

  INFO     2025-09-11 16:52:02,421 console_span_processor:62 telemetry: 16:52:02.365 [INFO]

           LiteLLM completion() model= gemini-2.0-flash; provider = vertex_ai

  INFO     2025-09-11 16:52:02,429 console_span_processor:39 telemetry: 16:52:02.422 [END]
  InferenceRouter.openai_chat_completion [StatusCode.OK]
           (55.76ms)

  INFO     2025-09-11 16:52:02,430 console_span_processor:48 telemetry: output: <async_generator object

           InferenceRouter.stream_tokens_and_compute_metrics_openai_chat at 0x7f0c3c29f4c0>

  ERROR    2025-09-11 16:52:02,991 __main__:253 server: Error executing endpoint route='/v1/openai/v1/responses' method='post':
   'ModelResponseStream'
           object has no attribute 'usage'

  INFO     2025-09-11 16:52:02,992 uvicorn.access:473 uncategorized: 10.131.0.115:50470 - "POST /v1/openai/v1/responses
  HTTP/1.1" 500
  INFO     2025-09-11 16:52:03,001 console_span_processor:39 telemetry: 16:52:02.994 [END]
  InferenceRouter.stream_tokens_and_compute_metrics_openai_chat
           [StatusCode.OK] (561.84ms)

  INFO     2025-09-11 16:52:03,002 console_span_processor:48 telemetry: chunk_count: 4

  INFO     2025-09-11 16:52:03,009 console_span_processor:39 telemetry: 16:52:03.004 [END] /v1/openai/v1/responses
  [StatusCode.OK] (689.96ms)
  INFO     2025-09-11 16:52:03,010 console_span_processor:48 telemetry: raw_path: /v1/openai/v1/responses

  INFO     2025-09-11 16:52:03,011 console_span_processor:62 telemetry: 16:52:02.992 [ERROR] Error executing endpoint
  route='/v1/openai/v1/responses'
           method='post': 'ModelResponseStream' object has no attribute 'usage'

  INFO     2025-09-11 16:52:03,012 console_span_processor:62 telemetry: 16:52:02.993 [INFO] 10.131.0.115:50470 - "POST
  /v1/openai/v1/responses
           HTTP/1.1" 500

Expected behavior

• Responses API should return a successful response object
• Telemetry should handle missing usage attributes gracefully

[mhdawson]

FYI, investigation assited by claude and have confirmed disabling telemetry allows calls to the Request AI to work when using gemini models.

[mhdawson]

Spending a while to see what it would take to get usage reporting this is the set of changes I currently have:

 1. LiteLLM Core - CustomStreamWrapper Usage Fix

  File: /usr/local/lib/python3.12/site-packages/litellm/litellm_core_utils/streaming_handler.py
  Component: LiteLLM Core Streaming InfrastructureDescription: Added logic to set .usage
  attribute in model_response_creator() method when send_stream_usage=True

  # In model_response_creator() method, after line ~1080:
  # USAGE FIX: Add usage attribute when send_stream_usage=True
  if hasattr(self, "send_stream_usage") and self.send_stream_usage and hasattr(self, "chunks"):
      # Calculate usage from accumulated chunks
      usage = calculate_total_usage(chunks=self.chunks)
      model_response.usage = usage

  2. LiteLLM VertexAI Provider - Stream Options Passing

  File: /usr/local/lib/python3.12/site-packages/litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
  Component: LiteLLM VertexAI ProviderDescription: Extract and pass
  stream_options to CustomStreamWrapper

  # Around line ~1285, before CustomStreamWrapper creation:
  # Extract stream_options from optional_params to enable usage tracking
  stream_options = optional_params.get("stream_options")

  streaming_response = CustomStreamWrapper(
      completion_stream=response,
      model=model,
      custom_llm_provider="vertex_ai",
      logging_obj=logging,
      stream_options=stream_options,  # Add this line
  )

  3. LlamaStack Responses API - Enable Usage Tracking

  File: /usr/local/lib/python3.12/site-packages/llama_stack/providers/inline/agents/meta_reference/responses/streaming.py
  Component: LlamaStack Agents - Meta Reference Responses
  APIDescription: Added stream_options={"include_usage": True} to openai_chat_completion call

  # Around line ~118, in the openai_chat_completion call:
  completion_result = await self.inference_api.openai_chat_completion(
      model=self.ctx.model,
      messages=messages,
      tools=self.ctx.chat_tools,
      stream=True,
      stream_options={"include_usage": True},  # Add this line
      temperature=self.ctx.temperature,
      response_format=self.ctx.response_format,
  )