[Feature] Enable remote inference endpoints for examples

[joshuayao]

Priority

P1-Stopper

OS type

Ubuntu

Hardware type

Xeon-GNR

Running nodes

Single Node

Description

Story for users

1. for Xeon users, users can try bigger models on remote endpoint with Gaudi while bigger models might not work well on Xeon
2. for Xeon users, if users want to try on Gaudi to understand the difference between Xeon and Gaudi, they can use remote endpoint
3. for Gaudi users, we assume that users have local Gaudi access to run GenAIExamples.
4. target Denvr and IBM for remote endpoints in the GenAIExamples

Enable remote inference endpoints for

• AgentQnA : Sin, Alex

  • AgentQnA - add support for remote server #1900
  • Agent: add support for remote server GenAIComps#1644
  • need CI test

• Productivity Suite : (Sri?)

  • Support denvr endpoints with Litellm. #2085
  • Enable remote inference non streaming response GenAIComps#1806
  • need CI test

• ChatQnA : (Sri?)

  • Support denvr endpoints with Litellm. #2085
  • Enable remote inference non streaming response GenAIComps#1806
  • need CI test

• DocSum : (Sri?)

  • Support denvr endpoints with Litellm. #2085
  • Enable remote inference non streaming response GenAIComps#1806
  • need CI Test

• CodeGen : (Sri?)

  • Support denvr endpoints with Litellm. #2085
  • Enable remote inference non streaming response GenAIComps#1806
  • need CI Test

• FinanceAgent : Alex

  • FinanceAgent - enable on Xeon, remote endpoint, and refactor tests #2032
  • need CI Test

• workflowExecAgent : Louie

  • Fix workexec agent docker build issues and enable LLM Remote Endpoint #2103
  • need CI Test

• CodeTrans : Alex

  • Codetrans: enable remote endpoints #2100
  • need CI test

• AudioQnA : Alex

  • Audioqna: fix input for TTS service, enable remote endpoints #2101
  • need CI test

Models are not supported in current public remote endpoint. ON HOLD for now.

• VideoQnA : (?)
• MulitmodalQnA : Alex
• VisualQnA: Alex

[louie-tsai]

@alexsin368 needs to be in assignees list

[joshuayao]

@alexsin368 needs to be in assignees list

@louie-tsai Ok, I will add @alexsin368 to the repo. Then, we can add him into the assignees list.
@alexsin368 Could you please accept the repo invitation in GitHub?

[alexsin368]

accepted, thanks