[Firebase AI] Add support for thought summaries

[andrewheard]

WIP - needs tests

[gemini-code-assist]

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

[google-oss-bot]

	1 Warning
⚠️	Did you forget to add a changelog entry? (Add #no-changelog to the PR description to silence this warning.)

Generated by 🚫 Danger

[gemini-code-assist]

Summary of Changes

This pull request introduces robust support for handling and exposing 'thought summaries' from Firebase AI models. It involves significant refactoring of the internal content representation to distinguish thought-related parts, updates to the public API for accessing these summaries, and modifications to content aggregation logic to ensure proper separation. Comprehensive tests have been added to verify the new functionality.

Highlights

• Core Data Model Enhancement: I've refactored the internal representation of ModelContent parts by introducing a new InternalPart struct with isThought and thoughtSignature properties. This allows the SDK to differentiate between regular content and the model's internal 'thought' processes, and to associate a unique signature with certain thought-related parts.
• Public API for Thought Summaries: I've added a new public property thoughtSummary: String? to GenerateContentResponse, providing a convenient way to access the aggregated text of thought parts from the model's response. The existing text property now specifically returns non-thought text.
• ThinkingConfig Expansion: The ThinkingConfig struct now includes an includeThoughts: Bool? parameter in its initializer. This allows developers to explicitly request or suppress thought summaries from the model, offering more granular control over the response content.
• Content Aggregation Logic Update: The aggregatedChunks function in Chat.swift has been updated to correctly separate and combine regular text parts from thought text parts, ensuring that thought summaries are not inadvertently merged with the main response text.
• Filtering Thought Parts from Other Types: I've updated the GenerateContentResponse to filter out thought parts when accessing functionCalls and inlineDataParts, ensuring that these properties only return relevant content types and not internal thought representations.
• Comprehensive Testing: I've added extensive unit and integration tests to validate the new thought summary functionality, covering both unary and streaming generateContent calls, and scenarios involving function calling, across Google AI and Vertex AI models.

Changelog

• FirebaseAI/Sources/Chat.swift
  • Refactored aggregatedChunks to correctly separate and aggregate 'thought' text from regular text, preventing their unintended combination.
  • Updated the internal parts array to use the new InternalPart type, which carries isThought and thoughtSignature metadata.
• FirebaseAI/Sources/GenerateContentResponse.swift
  • Added a new public property thoughtSummary: String? to access aggregated thought text.
  • Refactored the existing text property to use a new private helper text(isThought:) to retrieve only non-thought text.
  • Modified functionCalls and inlineDataParts properties to exclude parts marked as isThought.
• FirebaseAI/Sources/ModelContent.swift
  • Moved InternalPart to a top-level struct and added isThought: Bool? and thoughtSignature: String? properties.
  • Introduced InternalPart.OneOfData enum to encapsulate different data types within InternalPart.
  • Updated ModelContent's parts computed property and initializers to correctly map between public Part types and the new InternalPart structure, preserving thought metadata.
  • Updated Codable conformance for InternalPart and InternalPart.OneOfData to handle serialization/deserialization of thought metadata.
• FirebaseAI/Sources/Types/Internal/InternalPart.swift
  • Added isThought and thoughtSignature properties to ErrorPart for consistency, defaulting isThought to false.
• FirebaseAI/Sources/Types/Public/Part.swift
  • Introduced isThought: Bool as a new requirement for the Part protocol.
  • Updated TextPart, InlineDataPart, FileDataPart, FunctionCallPart, and FunctionResponsePart to conform to the Part protocol's new isThought requirement, and added internal _isThought and thoughtSignature properties to propagate this information.
• FirebaseAI/Sources/Types/Public/ThinkingConfig.swift
  • Added includeThoughts: Bool? property to ThinkingConfig to allow explicit control over thought summary inclusion in responses.
  • Updated the ThinkingConfig initializer to accept the new includeThoughts parameter.
• FirebaseAI/Tests/TestApp/Tests/Integration/GenerateContentIntegrationTests.swift
  • Updated generateContentThinking test to use the new ThinkingConfig with includeThoughts and added assertions for thought parts.
  • Added a new integration test generateContentThinkingFunctionCalling to verify thought summaries and signatures in function calling scenarios.
• FirebaseAI/Tests/Unit/GenerativeModelGoogleAITests.swift
  • Added unit tests testGenerateContent_success_thinking_thoughtSummary and testGenerateContent_success_thinking_functionCall_thoughtSummaryAndSignature to validate thought summary and signature parsing.
  • Added streaming unit tests testGenerateContentStream_successWithThoughtSummary and testGenerateContentStream_success_thinking_functionCall_thoughtSummary_signature.
• FirebaseAI/Tests/Unit/GenerativeModelVertexAITests.swift
  • Added unit tests testGenerateContent_success_thinking_thoughtSummary and testGenerateContentStream_successWithThinking_thoughtSummary for Vertex AI.

Activity

• The PR author, andrewheard, initially marked the PR as 'WIP - needs tests', but has since added comprehensive tests as seen in the patch.
• A Danger bot warning was posted regarding a missing changelog entry, which can be silenced by adding #no-changelog to the PR description.
• Reviewer paulb777 inquired about the exclusion of 'thoughts' from the history. andrewheard clarified that while an internal chat suggested exclusion, they decided to align with the Python SDK's approach of including them, leading to a change in the aggregation logic.
• paulb777 also questioned the use of an optional Bool for isThought. andrewheard provided a detailed explanation, citing potential backend validation changes and consistency with other SDKs (Python, TypeScript/JavaScript, Go, Java) where includeThoughts is also optional. They concluded in an offline discussion that the optional Bool is acceptable.