feat: [Orchestration] Embedding Convenience #562

rpanackal · 2025-08-29T14:04:58Z

Context

We would like to introduce convenience for embedding in orchestration module.

Feature scope:

New convenience classes embedding model, request and response convenience
- OrchestrationEmbeddingModel, OrchestrationEmbeddingRequest and OrchestrationEmbeddingResponse
- Embedding models added as per SAP Notes. All deprecated models skipped.
- Scenario test call returns inconsistent report on models compared to SAP Notes (ignored)
Add additional client method OrchestrationClient#embed(OrchestrationEmbeddingRequest)
E2E Embedding endpoint with masking configured
Support for float encoding only.
Return embedding as List<float[]>. API parity with OpenAI
- Optional: Generate model class with embedding as type List<float[]>. Setting generator config <useFloatArrays>true</useFloatArrays> seems ineffective. Needs investigation.

Definition of Done

Functionality scope stated & covered
Tests cover the scope above
~~Error handling created / updated & covered by the tests above~~
~~Aligned changes with the JavaScript SDK~~
Documentation updated [java] [1.12.0] Embedding in orchestration ai-sdk#213
Release notes updated

- client method for high level objects - Only support 'float' encoding format - Add javadoc - create convenience embedding response class - Add unit/integration tests - Add json payloads for req and res - Add e2e

rpanackal · 2025-09-01T13:14:53Z

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingResponse.java

+      final var bigDecimals = (Embedding.InnerBigDecimals) container.getEmbedding();
+      final var values = bigDecimals.values();
+      final float[] arr = new float[values.size()];
+      for (int i = 0; i < values.size(); i++) {
+        arr[i] = values.get(i).floatValue();


This is an unfortunate consequence of the openapi generator flag <useFloatArrays>true</useFloatArrays> not being supported for

oneOf: # works without the `oneOf` - type: array items: type: integer

This is concerning:

See memory footprint comparison

Memory Usage Breakdown

float[] array:

Each float uses exactly 4 bytes

Array overhead: ~12-16 bytes (object header)

Total for 1000 elements: ~4,012 bytes

List<BigDecimal> (assuming ArrayList):

ArrayList overhead: ~24 bytes + internal array overhead

Each BigDecimal object: ~40-48 bytes (object header + BigInteger + scale + precision)

Each BigInteger inside: ~24-32 bytes + int array for digits

Boxing overhead from list storage

Total for 1000 elements: ~80,000-100,000 bytes

Is this limitation originating from the openapi generator or our creators feature?

Do you think it's possible to fix it in our creators feature?

Is a follow-up BLI already considered?

openapi generator donot support this at all

our creator feature doesn't account for the combined use of USE_FLOAT_ARRAY feature.

PR on the way for fixing our creator feature. 😄

OpenAPI Generator PR: SAP/cloud-sdk-java#927

Is the above PR + Cloud SDK release a requirement? No right?

Not a requirement. Just a nice to have.

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingResponse.java

newtork · 2025-09-04T08:15:23Z

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingModel.java

+  /** Azure OpenAI Text Embedding 3 Small model */
+  public static final OrchestrationEmbeddingModel TEXT_EMBEDDING_3_SMALL =
+      new OrchestrationEmbeddingModel("text-embedding-3-small");
+
+  /** Azure OpenAI Text Embedding 3 Large model */
+  public static final OrchestrationEmbeddingModel TEXT_EMBEDDING_3_LARGE =
+      new OrchestrationEmbeddingModel("text-embedding-3-large");
+
+  /** Amazon Titan Embed Text model */
+  public static final OrchestrationEmbeddingModel AMAZON_TITAN_EMBED_TEXT =
+      new OrchestrationEmbeddingModel("amazon.titan-embed-text");
+
+  /** NVIDIA LLaMA 3.2 7B NV EmbedQA model */
+  public static final OrchestrationEmbeddingModel NVIDIA_LLAMA_32_NV_EMBEDQA_1B =
+      new OrchestrationEmbeddingModel("nvidia--llama-3.2-nv-embedqa-1b");


(Major/Question)

You are introducing another set of model constants we need to maintain. Can't we use existing class(es)?
What is the process of keeping this up-to-date?

My motivation of introducing new class was that embedding models do not share the same parameters as models in OrchestrationAiModel and are not valid for the chatCompletion calls. So, I wanted to avoid a user passing the wrong model to some endpoint.

And I am open to removing this class and extending OrchestrationAiModel. What do you think?

About maintenance, ideally, we can use an e2e (scenario) test to maintain this. But, unfortunately, this is pointless right now because SAP Notes and the api response are inconsistent.

We need the test for maintenance either way. But I would rather create a follow up BLI because this requires sync with orchestration.

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingRequest.java

newtork · 2025-09-04T08:22:01Z

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingRequest.java

+
+  /** Builder step for specifying text inputs to embed. */
+  @FunctionalInterface
+  public interface InputStep {


(Minor)

I'm not sure whether we used "Step" as "Builder" substitute already somewhere in the project. If not, please reconsider.

We have in fact used it once before in TemplateConfig. I actually like the current approach in how clean it is in usage and enforces both required arguments. But again, if you have a strong opinion, I will choose a static factory. Please confirm.

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingRequest.java

# Conflicts: # docs/release_notes.md # orchestration/src/test/java/com/sap/ai/sdk/orchestration/OrchestrationUnitTest.java

# Conflicts: # docs/release_notes.md # sample-code/spring-app/src/test/java/com/sap/ai/sdk/app/controllers/OrchestrationTest.java

…bedding-conv

newtork

LGTM

# Conflicts: # sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java

Jonas-Isr

LGTM

rpanackal added 2 commits August 28, 2025 10:41

Request convenience

20d72c3

Finishing up all major items

af8ecda

- client method for high level objects - Only support 'float' encoding format - Add javadoc - create convenience embedding response class - Add unit/integration tests - Add json payloads for req and res - Add e2e

rpanackal self-assigned this Aug 29, 2025

Add release notes and clean up

3e6a562

rpanackal added the please-review Request to review a pull-request label Sep 1, 2025

Merge branch 'main' into feat/orchestration/embedding-conv

8f3b16e

rpanackal commented Sep 1, 2025

View reviewed changes

CharlesDuboisSAP reviewed Sep 3, 2025

View reviewed changes

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingResponse.java Show resolved Hide resolved

newtork reviewed Sep 4, 2025

View reviewed changes

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingRequest.java Show resolved Hide resolved

newtork reviewed Sep 4, 2025

View reviewed changes

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingRequest.java Outdated Show resolved Hide resolved

newtork reviewed Sep 4, 2025

View reviewed changes

orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationEmbeddingRequest.java Outdated Show resolved Hide resolved

rpanackal and others added 8 commits September 4, 2025 14:29

Remove enum and remove redundant test

d7ea81c

Merge branch 'main' into feat/orchestration/embedding-conv

059339c

# Conflicts: # docs/release_notes.md # orchestration/src/test/java/com/sap/ai/sdk/orchestration/OrchestrationUnitTest.java

spec update +maxRetries and +timeout

f9384fa

model name correction

8e0cd53

Merge branch 'main' into feat/orchestration/embedding-conv

66bd11f

# Conflicts: # docs/release_notes.md # sample-code/spring-app/src/test/java/com/sap/ai/sdk/app/controllers/OrchestrationTest.java

Adjust since versions

3cc5581

Fix minor naming issues

2b6f8fe

Merge remote-tracking branch 'origin/main' into feat/orchestration/em…

f16f752

…bedding-conv

newtork previously approved these changes Sep 22, 2025

View reviewed changes

Jonas-Isr added 2 commits October 2, 2025 13:23

Merge branch 'main' into feat/orchestration/embedding-conv

ba04133

# Conflicts: # sample-code/spring-app/src/main/java/com/sap/ai/sdk/app/services/OrchestrationService.java

mini fixes

6e16516

Jonas-Isr dismissed newtork’s stale review via 6e16516 October 2, 2025 11:26

Jonas-Isr approved these changes Oct 2, 2025

View reviewed changes

Jonas-Isr merged commit 4cb0d53 into main Oct 2, 2025
7 checks passed

Jonas-Isr deleted the feat/orchestration/embedding-conv branch October 2, 2025 11:32

feat: [Orchestration] Embedding Convenience #562

feat: [Orchestration] Embedding Convenience #562

Uh oh!

Conversation

rpanackal commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Feature scope:

Definition of Done

Uh oh!

rpanackal Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

newtork Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Memory Usage Breakdown

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

newtork Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

newtork left a comment

Choose a reason for hiding this comment

Uh oh!

Jonas-Isr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rpanackal commented Aug 29, 2025 •

edited

Loading

rpanackal Sep 1, 2025 •

edited

Loading

newtork Sep 4, 2025 •

edited

Loading

newtork Sep 4, 2025 •

edited

Loading