FinanceAgent - enable on Xeon, remote endpoint, and refactor tests #2032

alexsin368 · 2025-06-04T21:22:21Z

Description

Enable support for running OpenAI models on Xeon
Enable remote endpoints for the LLMs on the agents only (docsum and dataprep will still run LLM locally)
Reorganize set_env.sh environment variables
Refactor tests: since Xeon test steps are similar to Gaudi's, breaking out the tests to make it easier to add new test steps or new tests for hardware

Issues

#1973

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

None

Tests

Added new test script.
Verified FinanceAgent is running on the UI.

Signed-off-by: alexsin368 <[email protected]>

github-actions · 2025-06-04T21:22:43Z

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

None

for more information, see https://pre-commit.ci

FinanceAgent/docker_compose/intel/hpu/gaudi/README.md

FinanceAgent/docker_compose/intel/hpu/gaudi/set_env.sh

MSCetin37 · 2025-06-04T23:57:24Z

FinanceAgent/docker_compose/intel/hpu/gaudi/README.md

 - tei-embedding-serving
 - redis-vector-db
 - redis-kv-store
 - dataprep-redis-server-finance
 - finqa-agent-endpoint
 - research-agent-endpoint
- docsum-vllm-gaudi
+- docsum-vllm-xeon


Could you double-check this? DocSum is using the same LLM models as the agents and is running on VLLM-Gaudi. This is an HPU compose, not a Xeon-based one.

Yes, originally DocSum and Dataprep use the same LLM models as the agents. What I've done is separate them with two models: one for agents, and another for DocSum/Dataprep.

For DocSum, it can run with vLLM on Xeon, and I've changed the LLM model for this to meta-llama/Llama-3.1-8B-Instruct. This 8B parameter will run fine on Xeon.

I haven't decided yet whether or not I want the remote endpoint/OpenAI model to be the LLM for the DocSum and Dataprep microservices. For simplicity, I'd leave them out for now.

FinanceAgent/README.md

MSCetin37 · 2025-06-05T17:29:16Z

FinanceAgent/tests/step2_start_services.sh

+        "xeon")
+            echo "==================== Start all services for Xeon ===================="
+            docker compose -f $WORKPATH/docker_compose/intel/cpu/xeon/compose_openai.yaml up -d
+            ;;


It might require some buffer time for all services to initialize and become ready.

Added check for the docsum-vllm-xeon service to be up and running. This should be the bottleneck and I've given it about 33 minutes, which is more than enough for loading an 8B parameter model.

FinanceAgent/docker_compose/intel/cpu/xeon/README.md

FinanceAgent/tests/step2_start_services.sh

Signed-off-by: alexsin368 <[email protected]>

…m/alexsin368/GenAIExamples into finance-agent-remote-endpoint-new

alexsin368

Addressed comments, but still need to fix test for Xeon.

louie-tsai

The PR looks confusing to me. Even we enable remote endpoint, vLLM is still running on Gaudi for Llama3 70B model. For Xeon recipe, do we move any micro-service from Gaudi to Xeon comparing to Gaudi recipe? if no architecture changes, why we claim a new Xeon support since no new support to move running from Gaudi to Xeon?

louie-tsai · 2025-06-06T16:42:08Z

FinanceAgent/README.md

+| Hardware                           | Deployment Mode                                                                             | Guide Link                                                               |
+| :--------------------------------- | :------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------- |
+| Intel® Gaudi® AI Accelerator     | Single Node (Docker)                                                                        | [Gaudi Docker Compose Guide](./docker_compose/intel/hpu/gaudi/README.md) |
+| Intel® Xeon® Scalable processors | Single Node (Docker) [Xeon Docker Compose Guide](./docker_compose/intel/cpu/xeon/README.md) |


I assume that those hardware platforms need to run LLama 70B with good accuracy. Do we plan to run Llama 70B on Xeon ?

See my set_env.sh. Xeon will run Llama 8B.

Llama 8b failed on AgentQnA accuracy check results. does Llama 8b give right answers for FinanceAgent?

Actually, the agent microservice will run with gpt-4o-mini-2024-07-18, the same as AgentQnA. Llama-3.1-8B-Instruct will be used for dataprep and docsum. Using the same models as other GenAIExamples.

louie-tsai · 2025-06-06T16:44:42Z

FinanceAgent/docker_compose/intel/cpu/xeon/README.md

@@ -0,0 +1,202 @@
+# Deploy Finance Agent on Intel® Xeon® Scalable processors with Docker Compose


could you remind me what micro-service we run on different architectures by introducing this PR? for remote endpoint, we still run on Gaudi not Xeon.

Agent microservice: remote endpoint (could be Xeon or Gaudi)
DocSum: vLLM on Xeon
Dataprep: Xeon

if we claim Xeon support for FinanceAgent, make sure that you have right responses from different models on Xeon. it is not the case for AgentQnA, and we need 70B for AgentQnA. for xeon folder, we probably need to focus on Xeon and have one microservice running on Xeon instead of Gaudi comparing to Gaudi folder.

…, fix python cmd Signed-off-by: alexsin368 <[email protected]>

louie-tsai · 2025-06-12T18:06:19Z

tag @louie-tsai to track the PR

…m/alexsin368/GenAIExamples into finance-agent-remote-endpoint-new

alexsin368 added 9 commits June 3, 2025 17:06

initial commit to enable remote endpoints

a9c1716

Signed-off-by: alexsin368 <[email protected]>

fix typo on file name

1f48bcb

Signed-off-by: alexsin368 <[email protected]>

add common-env

f12c122

Signed-off-by: alexsin368 <[email protected]>

fix environment

68b824d

Signed-off-by: alexsin368 <[email protected]>

correct path to top .set_env.sh file

7408871

Signed-off-by: alexsin368 <[email protected]>

explicitly set env variables for agent microservices

c3f4e9d

Signed-off-by: alexsin368 <[email protected]>

fix typos, initial commit for xeon test

f6f123a

Signed-off-by: alexsin368 <[email protected]>

reorganize env variables

62e3fbf

Signed-off-by: alexsin368 <[email protected]>

refactor tests for Gaudi and Xeon

de7fe5a

Signed-off-by: alexsin368 <[email protected]>

alexsin368 requested review from louie-tsai, yinghu5 and MSCetin37 June 4, 2025 21:22

alexsin368 requested review from ashahba, lkk12014402, minmin-intel, rbrugaro and XinyuYe-Intel as code owners June 4, 2025 21:22

[pre-commit.ci] auto fixes from pre-commit.com hooks

1c60777

for more information, see https://pre-commit.ci

alexsin368 mentioned this pull request Jun 4, 2025

[Feature] Enable remote inference endpoints for examples #1973

Open

35 tasks

MSCetin37 reviewed Jun 4, 2025

View reviewed changes

FinanceAgent/docker_compose/intel/hpu/gaudi/README.md Show resolved Hide resolved

MSCetin37 reviewed Jun 4, 2025

View reviewed changes

FinanceAgent/docker_compose/intel/hpu/gaudi/set_env.sh Show resolved Hide resolved

MSCetin37 reviewed Jun 4, 2025

View reviewed changes

MSCetin37 reviewed Jun 5, 2025

View reviewed changes

FinanceAgent/README.md Show resolved Hide resolved

MSCetin37 reviewed Jun 5, 2025

View reviewed changes

FinanceAgent/docker_compose/intel/cpu/xeon/README.md Show resolved Hide resolved

MSCetin37 reviewed Jun 5, 2025

View reviewed changes

FinanceAgent/docker_compose/intel/cpu/xeon/README.md Show resolved Hide resolved

MSCetin37 reviewed Jun 5, 2025

View reviewed changes

FinanceAgent/tests/step2_start_services.sh Show resolved Hide resolved

alexsin368 added 2 commits June 5, 2025 13:27

address comments, added check for docsum-vllm-xeon in test

bc7f7c4

Signed-off-by: alexsin368 <[email protected]>

Merge branch 'finance-agent-remote-endpoint-new' of https://github.co…

234f931

…m/alexsin368/GenAIExamples into finance-agent-remote-endpoint-new

alexsin368 commented Jun 5, 2025

View reviewed changes

louie-tsai reviewed Jun 6, 2025

View reviewed changes

make test _ because lack of OPENAI_API_KEY means cannot test in CI/CD…

dfe01b0

…, fix python cmd Signed-off-by: alexsin368 <[email protected]>

alexsin368 closed this Jun 16, 2025

alexsin368 force-pushed the finance-agent-remote-endpoint-new branch from dfe01b0 to 6ebae99 Compare June 16, 2025 23:14

Merge branch 'finance-agent-remote-endpoint-new' of https://github.co…

eae522a

…m/alexsin368/GenAIExamples into finance-agent-remote-endpoint-new

alexsin368 reopened this Jun 16, 2025

		@@ -0,0 +1,202 @@
		# Deploy Finance Agent on Intel® Xeon® Scalable processors with Docker Compose

FinanceAgent - enable on Xeon, remote endpoint, and refactor tests #2032

Are you sure you want to change the base?

FinanceAgent - enable on Xeon, remote endpoint, and refactor tests #2032

Uh oh!

Conversation

alexsin368 commented Jun 4, 2025

Description

Issues

Type of change

Dependencies

Tests

Uh oh!

github-actions bot commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

Scanned Files

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexsin368 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

louie-tsai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

louie-tsai commented Jun 12, 2025

Uh oh!

Uh oh!

github-actions bot commented Jun 4, 2025 •

edited

Loading

alexsin368 left a comment •

edited

Loading