Skip to content

Commit ef277b9

Browse files
louie-tsailouie-tsai
committed
Update README.md
Signed-off-by: Tsai, Louie <[email protected]>
1 parent e57b423 commit ef277b9

File tree

2 files changed

+18
-89
lines changed

2 files changed

+18
-89
lines changed

‎WorkflowExecAgent/README.md

Lines changed: 0 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -79,98 +79,9 @@ The table below lists currently available deployment options. They outline in de
7979

8080
## Validated Configurations
8181

82-
<<<<<<< HEAD
8382
| **Deploy Method** | **Hardware** |
8483
| ----------------- | ------------ |
8584
| Docker Compose | Intel Xeon |
86-
=======
87-
![image](https://github.com/user-attachments/assets/969fefb7-543d-427f-a56c-dc70e474ae60)
88-
89-
## Microservice Setup
90-
91-
### Start Agent Microservice
92-
93-
Workflow Executor will have a single docker image.
94-
95-
(Optional) Build the agent docker image with the most latest changes.
96-
By default, Workflow Executor uses public [opea/vllm](https://hub.docker.com/r/opea/agent) docker image if no local built image exists.
97-
98-
```sh
99-
export WORKDIR=$PWD
100-
git clone https://github.com/opea-project/GenAIComps.git
101-
cd GenAIExamples//WorkflowExecAgent/docker_image_build/
102-
git clone https://github.com/opea-project/GenAIExamples.git
103-
docker compose -f build.yaml build --no-cache
104-
```
105-
106-
<details>
107-
<summary> Using Remote LLM Endpoints </summary>
108-
When models are deployed on a remote server, a base URL and an API key are required to access them. To set up a remote server and acquire the base URL and API key, refer to <a href="https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/enterprise-ai.html"> Intel® AI for Enterprise Inference </a> offerings.
109-
110-
Set the following environment variables.
111-
112-
- `llm_endpoint_url` is the HTTPS endpoint of the remote server with the model of choice (i.e. https://api.inference.denvrdata.com). **Note:** If not using LiteLLM, the second part of the model card needs to be appended to the URL i.e. `/Llama-3.3-70B-Instruct` from `meta-llama/Llama-3.3-70B-Instruct`.
113-
- `llm_endpoint_api_key` is the access token or key to access the model(s) on the server.
114-
- `LLM_MODEL_ID` is the model card which may need to be overwritten depending on what it is set to `set_env.sh`.
115-
116-
```bash
117-
export llm_endpoint_url=<https-endpoint-of-remote-server>
118-
export llm_endpoint_api_key=<your-api-key>
119-
export LLM_MODEL_ID=<model-card>
120-
```
121-
122-
</details>
123-
124-
Configure `GenAIExamples/WorkflowExecAgent/docker_compose/.env` file with the following. Replace the variables according to your usecase.
125-
126-
```sh
127-
export SDK_BASE_URL=${SDK_BASE_URL}
128-
export SERVING_TOKEN=${SERVING_TOKEN}
129-
export HF_TOKEN=${HF_TOKEN}
130-
export llm_engine=vllm
131-
export llm_endpoint_url=${llm_endpoint_url}
132-
export api_key=${llm_endpoint_api_key:-""}
133-
export ip_address=$(hostname -I | awk '{print $1}')
134-
export model=${LLM_MODEL_ID:-"mistralai/Mistral-7B-Instruct-v0.3"}
135-
export recursion_limit=${recursion_limit}
136-
export temperature=0
137-
export max_new_tokens=1000
138-
export WORKDIR=${WORKDIR}
139-
export TOOLSET_PATH=$WORKDIR/GenAIExamples/WorkflowExecAgent/tools/
140-
export http_proxy=${http_proxy}
141-
export https_proxy=${https_proxy}
142-
```
143-
144-
> Note: SDK_BASE_URL and SERVING_TOKEN can be obtained from Intel Data Insight Automation platform.
145-
> For llm_endpoint_url, both local vllm service or an remote vllm endpoint work for the example.
146-
147-
Launch service by running the docker compose command.
148-
149-
```sh
150-
cd $WORKDIR/GenAIExamples/WorkflowExecAgent/docker_compose
151-
docker compose -f compose.yaml up -d
152-
```
153-
154-
### Validate service
155-
156-
The microservice logs can be viewed using:
157-
158-
```sh
159-
docker logs workflowexec-agent-endpoint
160-
```
161-
162-
You should be able to see "HTTP server setup successful" upon successful startup.
163-
164-
You can validate the service using the following command:
165-
166-
```sh
167-
curl http://${ip_address}:9090/v1/chat/completions -X POST -H "Content-Type: application/json" -d '{
168-
"query": "I have a data with gender Female, tenure 55, MonthlyAvgCharges 103.7. Predict if this entry will churn. My workflow id is '${workflow_id}'."
169-
}'
170-
```
171-
172-
Update the `query` with the workflow parameters, workflow id, etc based on the workflow context.
173-
>>>>>>> 5b864ae7 (Enable Remote Endpoints for LLM using Intel Enterprise Inference)
17485

17586
## Roadmap
17687

‎WorkflowExecAgent/docker_compose/intel/cpu/xeon/README.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,24 @@ export temperature=0
6060
export max_new_tokens=1000
6161
```
6262

63+
<details>
64+
<summary> Using Remote LLM Endpoints </summary>
65+
When models are deployed on a remote server, a base URL and an API key are required to access them. To set up a remote server and acquire the base URL and API key, refer to <a href="https://www.intel.com/content/www/us/en/developer/topic-technology/artificial-intelligence/enterprise-inference.html"> Intel® AI for Enterprise Inference </a> offerings.
66+
67+
Set the following environment variables.
68+
69+
- `llm_endpoint_url` is the HTTPS endpoint of the remote server with the model of choice (i.e. https://api.inference.denvrdata.com). **Note:** If not using LiteLLM, the second part of the model card needs to be appended to the URL i.e. `/Llama-3.3-70B-Instruct` from `meta-llama/Llama-3.3-70B-Instruct`.
70+
- `llm_endpoint_api_key` is the access token or key to access the model(s) on the server.
71+
- `LLM_MODEL_ID` is the model card which may need to be overwritten depending on what it is set to `set_env.sh`.
72+
73+
```bash
74+
export llm_endpoint_url=<https-endpoint-of-remote-server>
75+
export llm_endpoint_api_key=<your-api-key>
76+
export LLM_MODEL_ID=<model-card>
77+
```
78+
79+
</details>
80+
6381
### Deploy the Services Using Docker Compose
6482

6583
For an out-of-the-box experience, this guide uses an example workflow serving API service. There are 3 services needed for the setup: the agent microservice, an LLM inference service, and the workflow serving API.

0 commit comments

Comments
 (0)